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Editorial 


As befits the role of an American mathematician in modern society, I have nothing practical 
to say about information theory. However the devotees of this theory may be interested in 
the reactions of an outsider who has followed some of its development. 

In spite of all the suggestive work by Wiener, Shannon, and their successors, the main thing 
that strikes an outsider is that there are so few theoretical results. In fact almost every time 
a writer proves an assertion connecting the capacity of a channel with the entropy of a source, 
his paper P,, is succeeded by a paper P,,,, which, instead of generalizing or extending the results 
of P,,, is devoted to pointing out and correcting some defect or insufficiency in it. The paper 
P.,.;, in its turn, receives the same harsh treatment, and so on. Moreover, in this presumably 
convergent process of purging and purifying, the theorems become more and more attenuated 
and inapplicable as their hypotheses become more restrictive. 

Even more extraordinary is the fact that this process of organizing what seems to be the 
very basis of the subject seems to have no effect whatever on its applications! Can it be that 
the existence of a mathematical basis is irrelevant, and that the basic principle is the very 
idea that there is a context in which the word “information” is accepted by general agreement 


and used in an intuitive way, and that no more is needed? 


J. L. Doos 


{ IRE TRANSACTIONS ON INFORMATION THEORY 


Marc 


Correlation and Delay Line Attenuation 


MELVIN J. JACOBSON} 


Summary—tThe effect of delay line attenuation on the output 
of a correlator is studied for the case where the attenuation in 
db to within a frequency independent loss varies linearly with 
delay and as the square root of frequency. It is shown how attenua- 
tion affects the output signal-noise ratio of the correlator, increasing 
it for some signal spectrum shapes and decreasing it for others. 
Upper bounds on the increase and decrease are computed and a 
sufficient condition on the input signal spectrum is established 
which, when satisfied, assures a degradation in signal-noise ratio. 
Also examined is the effect of a frequency independent gain inserted 
to compensate for the delay line attenuation. It is shown how this 
gain may be chosen in order to give uniform system output noise 
over all values of delay. 


I. Norarion 


f = frequency 

fe = upper frequency accepted by system 

fi = lower frequency accepted by system 

r = bandwidth ratio f./f,; 

s(f) = signal power density 

n(f) = noise power density 

T = relative delay in signal in the two system channels 


| 


g inserted delay 


S = average input signal power 
N = average input noise power 
k = delay line constant 


01, ly = kéf,?, kéfy? 


Il. InrrRropuction 


T is well known that a system employing a cross- 
correlation device may be used to determine the 
presence of a signal in noise. A portion of a correla- 

tion system containing a delay line is shown in Fig. 1.’ 
In channel 1, s(t) is the signal voltage, n,(t) is an un- 
desired noise voltage, and ¢ is a measure of time. In 
channel 2, an undesired noise voltage n.(t) is present 
together with the desired signal voltage s(t + 7). Note 
that the signal in channel | is delayed relative to that in 
channel 2 by the time 7. The information in channel 2 is 
now delayed for a time & after which the voltages in the 
two channels enter the correlator where they are multiplied 
together and averaged over a finite time interval to give 
the output of the system. When é takes on the value 7, 
the signal at both inputs to the correlator will be the same 
and maximum signal cross-correlation will result. The 
delay line may be switched to channel | to determine the 


* Manuscript received by the PGIT, March 24, 1958. The work 
leading to this paper was supported in part by Office of Naval 
Research Contract Nonr-591(09). 

z Leeee of Mathematics, Rensselaer Polytechnic Inst., Troy, 

1 For another analysis of a similar system, see P. E. Green, 
Jr., “The output signal-to-noise ratio of correlation detectors,’ 
IRE Trans. oN INForMATION THEORY, vol. IT-3, pp. 10-18; 
March, 1957. 


CHANNEL | CHANNEL 2 
s(t) s(t+T) 
+n, (t) +No(t) 
DELAY § 


CORRELATOR 


OUT PUT 


Fig. 1—Portion of a delay line correlation system. 


presence of a signal which is delayed in channel 2 relativ: 
to channel 1. 

It is the purpose of this paper to investigate hov 
attenuation in the delay line influences the effectivenes 
of the correlation system in determining the presence of : 
signal in noise. In particular, we shall be concerned witl 
an examination of the output signal-noise ratio and th 
output noise of the system described above when the 
delay line attenuation in db, to within a frequency in 
dependent loss, varies linearly with delay and as th 
square root of frequency. Such an attenuation will resul 
if the input frequencies are in the low megacycle regior 
and if the delay line makes use of uniformly dissipativs 
lumped constant delay networks having air core solenoic¢ 
coils designed for minimum loss.” * Of course, this restric 
tion on the input frequencies to the delay line and corre 
lator does not require that the signal and noise be in thi 
low-megacycle region initially. 

In the following sections it will be assumed that th 
noise in the two system channels have the same powe 
spectra but do not correlate. Further, it will be assumec 
that the signal and noise do not correlate and that both 
the signal and noise possess the ergodic and gaussiai 
properties. 


Ill. Tue [peat System 


When a correlation system contains a loss-free dela: 
mechanism, the signal and noise power spectra in the tw 
system channels are the same, and the representation 
for the mean and variance of the output are well known. 
In particular, the mean system output is given by 


2 J. F. Blackburn, “Components Handbook,’ M. I. T. Rac 
Lab. Ser., McGraw-Hill Book Co., New York, N. Y., ch. 6; 194! 

3$. Ramo and J. R. Whinnery, ‘‘Fields and Waves in Moder 
Radio,’ John Wiley and Sons, Inc., New York, N. Y., sec. 6.1. 
1944. 

4A. T. Starr, “Electric Circuits and Wave Filters,” Pitma 
Publ. Corp., New York, N. Y., pp. 124-127; 1944. 

5 J. J. Faran, Jr. and R. Hills, Jr., “Correlators for Sign 
Reception,’’ Acoust. Res. Lab., Harvard University, Cambridg 
Mass., Tech. Memo. 27; 1952. 
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Rule — 1) = | s(f) cos 2nfl — 2) af (1) 


id the variance is given by 


1 fee) 
-~ 2RC io (fei 4.2) Fis ae) 


oe het fa) Woe — €or et) de. (2) 


hen the correlator contains an RC averaging network 
id RC(f. — f,) > 1. The quantities R,, and Ry» are the 
itocorrelations of the outputs of channels 1 and 2 
spectively, and R,, and R,, are cross correlations of the 
itputs of the two channels. or the particular case of 
eat interest where the input signal power is much less 
an the input noise power, we may take 


Rae) = h(x) ~ AG cos Qrfx df (3) 


es 


id neglect cross-correlation terms in (2) to get 


Z 1 ve 
ST ORC ie ub 


we also suppose that the noise power spectrum is flat 
fer fi < f < fe, then 


nf) cos 2arfx ar} ax. (4) 


N 
uf) = a 5 
nf) emer (5) 
id the variance reduces further to 
2 eee Verret 
AP RR CG 6) 


the noise spectrum is not flat, one can always equalize 
)make it so. 

One accepted measure of the quality of this type of 
stem is the maximum value of the output signal-noise 
ywer ratio, defined as the ratio of the square of (1) to 
) with £ — 7 equated to zero. When  — 7 = O, 


RO) = [sh af= 8 (7) 


id the output signal-noise ratio is given by 


(3) PROG, = ple). (8) 


LV. IncLusIon or ATTENUATION 


We shall now examine the effect of delay line attenu- 
ion on correlator output signal-noise ratio when the 
squency range of the input to the delay line is in the 
w megacycle region. Of course, this does not necessarily 
strict the frequencies initially accepted by the system 
lie in this range. With the type of delay line described 
Section I we may assume that, except for a flat (fre- 
lency independent) loss, the attenuation in the line in 
is equal to 8.69kéf'’” where k is a delay line constant. 
1en the attenuation in nepers is equal to kéf'””. 

We begin by assuming that the noise voltage v; at the 
put to the delay line may be expanded in a I ourier 


series over the time interval 0 < t < T to give 


2rmt 


ee: b, Sin (9) 


S 


A) ee Sor ee 


m=1 


where the a,, and b,, are independent variables distributed 
normally with zero mean and the same variance, and 
m/T is the frequency of the m’th sinusoidal component 
of the noise.” If this noise voltage is now passed through 
the delay line, the output to within a flat loss is given by 


= vie > E cos Pamee— 8) £) 


m= 


v(t) = 


Pam — 9 | exp [—k&(m/T)'”] (10) 


foré <t < 7 + £&. If we now form the product of vo(t) 
and v(t — 2x), average over time, and let 7’ become 


infinite, it is found that the autocorrelation of the noise 
in the delayed channel is given by 


+ b,, sin 


i n(f) exp (—2kéf'’””) cos 2rfx df. (11) 


The autocorrelation of the noise in the undelayed channel 
is given by (3) and, analogous to (4), the variance of the 
system output is given by 


et sae A | it ny Cunoa re ar] 


| [nop exp (249%) cos 2ate af | - 


to within a flat loss. When the input noise spectrum is 
flat, the first inner integral is easily evaluated and (12) 
may be written as 


2 N? = 1/2 
aeRO i 


f sin 2nfow - Soe cos 2rfx dx df. (13) 
The integral over x is equal to 7’ so that 
3 N’ as fone ne ! 
> Gaag, Sah] Te okt) eRe) 


In a similar fashion, it can be shown that the mean system 
output is given by 


0) = | s(f) exp (—kef®) af (15) 


when £ — 7 = 0 and the output signal-noise ratio may be 


written as 


(S) = Ret, mol a exp (—kéf'”) at] 


N N fe Sy 
[eso (—28et” ay 


6S. O. Rice, “Mathematical analysis of random noise,” Bell Sys. 
Pech. J., vol. 23, pp. 282-332; July, 1944. (Sec. 2.3.) 

TO, 13% DeHaan, “Nouvelles Tables D’Intégrales Définies,’’ 
G. E. Strechert and Co., New York, N. Y.; 1939. (Table 151.) 


6 IRE TRANSACTIONS ON INFORMATION THEORY 


It can be seen at this point that a frequency independent 
loss in the delay line has no effect upon output signal-noise 
ratio, since the appropriate factors, if included in (14) and 
(15), would cancel in (16). 

In the two sections to follow, we shall be interested in 
the ratio of the output signal-noise ratio for the attenua- 
tion case to that for the ideal case. From (8) and (16), this 
ratio 1s given by 


i | ih ; exp (—kef') ar 


[pees le (17) 
| exp (—2kéf'””) df 


2 


A value of D less than one will represent a degradation in 
output signal-noise ratio resulting from delay line attenu- 
ation, while a value of D greater than one will represent 
an improvement in signal-noise ratio. 

V. Constant SLOPE SIGNAL SPECTRA 


When a signal spectrum has a constant slope of 3a db 


per octave, the power density may be written as 
Gp) SiGe (18) 


where K must satisfy 


Kk (19) 


BEEN 
oe 


if the signal is to have average power S in the band 
fi <f <f,. If (18) and (19) are substituted into (17) and 
if f, — f, is given an integral representation, then we may 


write 
to Lo 2 
2 ae 
i x dx / ge * dx 
Z1 Ti 8 
Ze Le 2 
=) 2 
i xe” dx / e*) dx 
wvT1 Ee 


0 and this ex- 


D(a) = (20) 


When the signal spectrum is flat, a = 


pression reduces to 
Le 2 
| [ aoe ax | 
721 


DO) = — = (21) 
xe” dx / x dx 
Now, by Schwarz’s inequality for integrals,* 
Xe 7 Le . 2 
i fy ae a | = | Gi (Ey es) ax | 
ee / 2 aw | xe dx (22) 


so that D(O) < 1 forx, > x,. Thus, a delay line attenuation 
of the type being considered here will give rise to a 
degradation in output signal-noise ratio for all frequency 


*G. H. Hardy, J. E. Littlewood, and G. Polya, ‘“Inequalities,”’ 
Cambridge University Press, Cambridge, Mass., p. 132; 1934. 


March 
bands and inserted delays when the signal spectrum is 
flat over the band f, < f < fo. 

We shall now demonstrate that D(a) < D(b) when 
a > b. That is, 


Le Xe 2 
Ma ee ae 
/ x dx | ge * dx 
ri ri 
Lo Le 2 
ip xe” dx l ag" dx 
Di ¥2X1 


i) “4 al | oe a | 
Ke ry Eee F 


Lo Do 2 (23) 
i ter axl te a | 
This desired inequality may be written as 
/ ; Tien ee dx / ; peed dx 
- / en ax | Eee O: (24) 


Now, after appropriate manipulation (see Appendix), the 
quantity on the left may be written as 


al ; / (ey)? ** [aro — oy? le * — e "| dx dy. (28) 


This is an integral in the first quadrant of the x, y plane 
over a square having a diagonal coincident with the line 
y = x. The integrand is negative throughout this square, 
except on the diagonal where it is equal to zero. Hence 
the value of the integral is negative and the desired 
inequality is proved. The implication of this result is that 
degradation in output signal-noise ratio increases as the 
slope of a signal spectrum having constant db per octave 
slope increases. 

If D(a) is expanded in a Taylor series about x, = 0, it 
is found that 


4(r’” — 1) 


De) =1+| Sig =} 


OE DG ee) 
(2a + 3)r'?(r°** — 1) 


|e. + O@:) (26) 


fora # — 1. The coefficient of x, is negative for a > 0 
and positive fora < Oso that D < 1fora > O0andD > 1 
for a < 0. Thus, the output signal-noise ratio for a de- 
creasing (negative slope) signal spectrum is actually 
increased by delay line attenuation when 8.69x,—the 
attenuation in db at the upper frequency of the acceptance 
band—is sufficiently small. . 

For decreasing as well as increasing spectra, there is a 
steadily increasing degradation in output signal-noise 
ratio as x Increases since, for 2, large, 


A(a + 1)? — Ir” 


D(a) oa Cc a 1)’2, 


(27) 


fora # — 1 and 
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g. 2—Degradation in output signal-noise ratio vs delay 
attenuation at upper frequency. Flat signal spectrum. 
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g. 3—Degradation in output signal-noise ratio vs delay line 
ittenuation at upper frequency. 3 db per octave signal spectrum. 


Ar — 1)r'” 


an 


(28) 


as Ihe 
The degradation D is plotted in db vs 8.692z,., the 
tenuation in db at the upper frequency fz, in Figs. 2, 
and 4. These figures correspond to a = 0, 1, and —1 
‘signal spectra having constant slopes of 0, 3, and —3 
9 per octave. In each figure D is plotted for r = 2, 4, and 
these values corresponding to bandwidths of 1, 2, and 
octaves. Curves are also plotted for the limiting cases 
— 1landr— o. Note that the curves of lig. 4 show a 
sgative degradation or an improvement in output signal- 
yise ratio relative to the loss-free case when x, 1s suf- 
‘iently small. 


VI. Morr GENERAL SPECTRA 


Since attenuation is least for small frequencies and 
eatest for large frequencies, minimum degradation will 
cur when the signal is represented by a line spectrum 
f = fi while maximum degradation will result when 
e signal spectrum is a line at f = fo. Letting f approach 
in (15) gives 


Te Oe=nse ot (29) 
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ATTENUATION AT 


F, (db) 


Fig. 4—Degradation in output signal-noise ratio vs delay line 
attenuation at upper frequency. —3 db per octave signal spectrum. 


The variance given by (14) and the output signal-noise 
ratio of the ideal system given by (8) remain unchanged. 
From (8), (14), and (29) the minimum degradation is 


given by 
Dae = = es mee - (30) 
62 (2a, il) see eee a) 
Similarly, the maximum degradation is given by 
2 x eats 2 p 2h 2 
De i (% ae (31) 


og ee eee 


The maximum and minimum degradation are plotted in 
db as a function of 8.69x, in Fig. 5. Note that, for fixed r, 
the curves of Figs. 2, 3, and 4 are bounded by those of 
Fig. 5. 

From (30) and (31) it can be seen that the largest 
possible variation in the output signal-noise ratio of the 
delay line correlation system is given by 


10 log,) exp 2(x2 — 2) = 8.69(1 — 1/r'””)x, db. (32) 


This variation appears graphically in Fig. 6 as a function 
Oi Gs. 

In addition to the above results, it is also possible to 
establish a sufficient condition on the signal spectrum 
which, when satisfied, assures a degradation in output 
signal-noise ratio relative to that of the ideal system. 
From (17) it can be seen that D will be less than one if 


fs 2 
ge — 10 [82 ex (ner a | 


ze i ep (he ea es) 


If s(f) is a continuous function of f, then (33) will be 
satisfied if 


(fe - po) See [exp (nef) a] 


< [exp (2p) af (34) 
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Fig. 5—Maximum and minimum degradation in output signal-noise 
ratio vs delay line attenuation at upper frequency. 
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Fig. 6—Maximum possible variation in output signal-noise ratio 
vs delay line attenuation at upper frequency. 


Where Sax 18 the largest value of the signal power density 
between f, and f.. If we now note that 


S = (fe = ise 


where s,, 1s the average value of s(f) between f, and f, 
then (84) may be written as 


ante 1/2 ato 1/2 
| x dx | ae” dx 
< Le oe Ty 
Sa are 
we Oe 
Z1 


where D(O) is the degradation in output signal-noise ratio 
for a flat signal spectrum and is plotted in db in Fig. 2. 


(35) 


S 


Omax 


=D"'0) 66) 


VII. Ourrut Noise 


In Section IV an expression was developed for the 
variance of the output of the delay line correlation system. 
Eq. (14) indicates that the variance or output noise is a 
function of the inserted delay and, in fact, decreases as £ 
increases. In searching for a signal in noise it would be 
desirable to have a uniform output noise for all values of 
inserted delay, and it is the purpose of this section to 
indicate a method which will provide such a result. In 
particular, we shall assume that in addition to a gain 


March 


inserted to offset the flat loss of the delay line, a compen- 
sating gain is added which is equal to 8.69kéf, db where 
fo is a frequency lying between f; and f,. This compen- 
sation could be obtained by inserting an appropriate 
variable attenuator and amplifier following the delay line. 
It will be noted that this gain will not affect the results of 
the previous sections, since the inclusion of frequency 
independent loss or gain will not alter (17). 

With the insertion of the gain described above, the 
output noise of the correlator is given by 


2 fhe 
x N 


“exp 2ke(fi? — f) df. 7) 


7 ARC, — fi br, 
If we now require that o” not only be constant but equal 


to the output noise of the ideal system, then from (6) and 
(37) we have that 


fo 
N exp 2ke(f? — f'?) df = N. 


(ad: = 
The right side of this equation is the noise power at the 
input to the delay line and the left side is the noise power 
at the output of the amplifier. Hence, the above require- 
ments on the correlator output noise are analogous to the 
requirement that there be no loss in noise power in the 
delayed channel of the system. Evaluation of (38) gives 


sot 25 ne oy) 
(2x, + le ** — (2x, + Ie” 


(39) 


where x) = kéf?. For fixed values of the frequency range 
and inserted delay, the above expression gives the fre- 
quency fy at which unity gain should be required in order 
to have no loss in noise power in the delayed channel. 
This frequency is a function of € and therefore changes 
as the value of the inserted delay is changed. The relation 
between 8.697) and 8.692, as given by (39) appears 
graphically in Fig. 7 for various values of r. The figure 
may also be interpreted as giving the required amount of 
compensating gain 8.692, db. 


VIII. Conciusion 


In this paper, some effects of delay line attenuation on | 
the output of a correlation system have been examined | 
for the case where the input to the delay line is in the low | 
megacycle region and the line contains uniformly dissi- 
pative lumped constant delay networks having air core | 
solenoid coils designed for minimum loss. In this case the 
attenuation in db to within a flat loss, varies linearly with , 
delay and as the square root of frequency. The output of 
this system has been compared to that of an ideal system 
in which there is no attenuation in the delayed channel. 

It has been shown that the output signal-noise ratio of — 
the delay line system may be larger or smaller than that 
of the ideal system depending upon the shape of the input 
signal spectrum. Upper and lower bounds on the output 
ratio of the delay line system have been found. In addition 


& g 


COMPENSATING GAIN AT Fo 
S 


v 
6O B80 


(db) 


ig. 7—Compensating gain vs delay line attenuation at upper 
frequency for uniform output noise. 
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condition on the signal spectrum has been established 
hich, when satisfied, assures a degradation in output 
ignal-noise ratio resulting from delay line attenuation. 
‘hese results are independent of a frequency independent 
ompensating gain which might be added in the delayed 
hannel. It has also been shown that delay line attenuation 
ffects the output noise of the system, causing it to decrease 
s the amount of delay is increased. It is desirable to have 

constant output noise over delay, and it has been 
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demonstrated how this may be accomplished by inserting 
an appropriate compensating gain in the dealyed channel 
which is independent of frequency. 


APPENDIX 


The quantity on the left of (24) may be written in terms 
of double integrals as 


Le Ge i. Lo 

Za la 2b+ 201 ee tee 
/ / ae ty dx dy — / / iy Cm ayy 
Ly ry D1 Ur1 


et / / po ie we Cae Ss jg dx dy 


1 a A 2aar 26+ —£ ny 
ll / gy? Te"? — e") dg dy 


Interchanging x and y in the second integral and combining 
the two resulting double integrals gives 


ae / ; (au) 


As described in Section V, the value of this integral is 
necessarily negative since the integrand is negative 
throughout the region of integration, except along the 
line y = x where its value is zero. 


— to ies =e daeay ane) 


On the First Probability of Detection by a 


Radar Receiver System’ 


W. M. STONE, R. L. BROCK, ann K. J. HAMMERLET 


Summary—Expressions for the detection probabilities associated 
ith the output of filter-square law detector-filter radar receivers 
‘e presented for practical filter systems and with a slowly varying 
ayleigh distributed signal amplitude. 


the first probability of detection of a signal in 
the presence of noise using a filter-square law 


| ‘ AC AND Siegert’ and Emerson” have discussed 


* Manuscript received by the PGIT, June 11, 1958. 

+ Boeing Airplane Co., Seattle, Wash., and Oregon State College, 
orvallis, Ore. 

t Boeing Airplane Co., Seattle, Wash. 

1M. Kac and A. J. F. Siegert, “On the theory of noise in radio 
ceivers with square law detectors,” J. Appl. Phys., vol. 18, 
». 383-397; April, 1947. 

2 R. C. Emerson, ‘First probability densities for receivers with 
uare law detectors,’ J. Appl. Phys., vol. 24, pp. 1168-1176; 
~ptember, 1953. 


detector-filter system. The present authors’ have extended 
the theory in two ways: 1) the signal is assumed to be 
randomly modulated, and 2) the filters employed are 
physically realizable in the sense of zero response before 
zero time. This paper presents the principal results 
achieved. 

It is considered that a discrete sampling of a continuous 
signal in the presence of noise is performed at the system 
output. The criterion of performance is measured by the 
probability, P(x, y, 2), that a sampled value of the system 
output shall exceed a given bias level, where 2 is the 
normalized bias level, y is related to the ratio of band- 


3 W. M. Stone and R. L. Brock, “‘On the Probability of Detection 
with a Postdetection Filter,’ Boeing Airplane Company Document 
D-16921; 1955. 
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widths of the two filters, and z is the expected value of the 
signal to noise power ratio. 

Two tacit assumptions’ are usually made in the first 
probability theory for signal detection: 1) the threshold 
device gets only one “look” at the system output when 
the signal is present and 2) the signal has been present 
for so long a time that its buildup in the filters can be 
ignored and modulation such as that die to antenna 
scanning need not be considered. These assumptions are 
really contradictory. A better approach would be to 
consider the actual signal envelope shape and to determine 
the probability that the bias level will be exceeded at 
least once during the presence of the fleeting signal. A 
preliminary study along these lines has been accomplished 
by the present writers. 

The first of these two assumptions will usually lead to 
no significant error when a properly designed system is 
considered. In such a system the time constant of the 
second filter will be roughly equal to the signal duration 
so that, in general, the threshold device will get only one 
“look” at the signal as the radar beam scans across the 
target. 

To compensate for the error introduced by the second 
of these assumptions, the signal to noise ratio may be 
adjusted. When the second filter bandwidth and the input 
signal modulation envelope are known, the time function 
representing the signal presented to the threshold in the 
absence of noise is readily determined. The ratio of the 
peak value of this function to the peak value that would 
be reached with an audio filter of negligibly short time 
constant can be calculated. This ratio, together with the 
“law” of the detector, enables one to make the appropriate 
correction. A second correction to allow for a signal which 
is off-center in frequency is readily incorporated into the 
theory. 

The basic formulations of the receiver system are the 
same as those of Kac and Siegert and Emerson. Let the 
first filter possess the frequency response function, 


k > 1, 


Oo = ka, ) 


and let the second filter possess the response function, 
= 
Fe) = E + j | : (2) 


In dimensionless variables the system kernel of the 
pertinent homogeneous integral equation may be readily 
determined as 


2 


Di Cee Se Net oe 
g(t) y) = 57 cos kyla = ¥) 


—2y7(zt+y)p 2(2y-1)z2 

€ le = Ie 0) Oe 
—2y (x+y) 7 2(2y-1)y i w/w. (3) 
é le = LE Oya 


IA 
lA 


=e Middleton, “Statistical criteria for the detection of pulsed 
carriers 1n noise,”’ J. Appl. Phys., vol. 24, pp. 371-391; April, 1953. 


March 


In principle the system kernel is developable in a bilinear 
expansion of the set of orthonormal functions, 


“ds via (Aas 
A heniVies) 


a 2hyx, 
sin 2kyx, 


je ene 


OS. <2, Sot) a (4) 


but the orthonormal property holds only for relatively 
large values of ky. Eq. (4) is equivalent to (7.18) and 
(7.19) of Kac and Siegert. 

For greater generality the signal is assumed to be 
sinusoidal but of frequency not necessarily equal to the 
center frequency w, of the first filter, 


S() = A coswt. (5) 
Since wy >> w, it may be specified that 
| @ = wo |/o (6) 


is of order unity while (# + wo)/w, > 1. A set of functions 
of immediate interest is 


a= 


(oe) 


a! _»~| COS 2yax 
é Deine \f i 


Pea) = / dx, (7) 


i ae Qyax 
which is implicit in (5.18) of Kac and Siegert. 

Without further ado the Laplace transform of the 
probability density function associated with the output 
of the system takes the form 


ols, nya y) =; ols, ae 0) 
“exp E > (—1)*"(8y8)'B.la, »|. (8) 


where y is the normalized signal amplitude, ¢(s, y, 0) is 
the Laplace transform for the noise only case, and the B, 
are defined by 


— Rowe 5 jab es eeoe 
Ba, y) = »S = Mo tht (r,) ta, %) ) 
n=1 n ZY TH 


NE ee (9) 


At least the first four of the B, are obtainable in closed 
form. The probability that a sampled value of output, 
noise only input, exceeds a given bias level is obtained as 


1 iy ieee 
Wee 


which has been tabulated in a Boeing document’ for 
% = 1.5(0.1)3.5 and y = 0(1)40. Eq. (10) is equivalent to 
(57) of Siegert.” For signal amplitude a constant the only 
possibility of inverting the Laplace transform of (8) is to 
resort to an Edgeworth series. But it makes physical sense 
to assume that 7 is a Rayleigh distributed random variable. 
Such a distribution is defined by 


P(Xo, Y> 0) (10) 


_ °A. J. F. Siegert, “A systematic approach to a class of problems 
in the theory of noise and other random phenomena—Part II, 
examples,’ IRE Trans. on Inrormation Tueory, vol. IT-3, 
pp. 38-43; March, 1957. 
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—y?/2z 


1 
f(y) aa OES ) gy 0), 
0, y —.0; 


(11) 


ere z is the average signal to noise power ratio. The 
ective average signal to noise power ratio is 


3 (12) 


d, finally, the probability of detection for relatively 
rge y becomes 

(Qy /z;) (a7 eee 
T(2y)J 2, =i [2(2y/z,)'”"] 


= it 
== e ali 5 ay 
Zi w 2\(2y + 1) 


— 3 is eee ae 
BAOy ee 4) | : 


he series form would be useful only for z,; > 1. 
In similar fashion a system with a double-tuned LC 
st filter, 


— 2 —— a 
E (° : 2») as 2b) ° “| 
1 WO) 


+ E s c=) + 2bj° = ve | ook) 


Po, 2%) = (13) 


(w) = 


and a second filter specified by (2) leads to 


JAGR rea aga ay 


al 


feeumes dyer) Be zal 
Zi(4y + 4by + 1)(2by + 1) d 


Oa @;/W2, (15) 


a result not significantly different from (13). The prob- 
ability of detection for a Rayleigh distributed signal 
amplitude for Gaussian filters is developed with little 
difficulty. It is recognized that a Gaussian filter is not 
physically realizable in the sense previously defined. The 
moments for all three systems are also given in the refer- 
enced document.” 

Large y implies that the post-detection smoothing is 
very good, so that for a given signal strength the output 
of the second filter is proportional to the power through 
the first filter, with negligible variance about this value. 
If z, is assumed large the power present would be almost 
all signal power. Since the signal amplitude follows a 
Rayleigh distribution the signal power would be ex- 
ponentially distributed, as is indicated by the leading 
terms of the series in (33) and (14). 

The authors wish to express their appreciation for the 
many remarks and suggestions of the referee. 
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Full Decodable Code-Word Sets’ 


M. P. SCHUTZENBERGER} anv R. 8S. MARCUS# 


Summary—This paper considers further how the decodability 
condition imposes restrictions on a set of code words. A generating 
function is defined that describes the composition of the code 
words. The relation between the generating function and a “full” 
set of code words is found. This relation shows that the sum of 
arbitrary probabilities associated with the words of a full set must 
be one. A full set of code words is one to which no code word can 
be added and still keep the set decodable. It is also shown that 
a full set is ‘“completable.’”’ For a completable set of code words 
any string of symbols can be made into a sentence by adding a 
suitable prefix and a suffix. 


INTRODUCTION 


EVERAL authors have considered the restrictions 
S that are imposed on the set of code words by the 

decodability condition."-’? (A  code-word set is 
decodable if no string of symbols can be broken up into 
code words in more than one way.) Most of the results 
thus far have had to do with the lengths of the code words. 
This paper includes some conclusions relating to the more 
detailed composition of the code words. 

It is important to consider the composition of the code 
words, as well as their lengths, when the symbols are not 
of the same cost. For example, in the Morse code the dot 
is shorter in time duration than the dash. The less costly 
dot, therefore, should be used more frequently for ef- 
ficiency of information transmission. 

In particular, this paper defines a generating function 
that describes the composition of the code words. The 
relation between this function and a “full” set of code 
words is found. A full set of code words is one to which 
no code words can be added and still keep the set de- 
codable. It is also shown that a full set is “completable.”’ 
For a completable set of code words, any string of symbols 
can be made into a sentence by adding a suitable prefix 
and a suffix. 


* Manuscript received by the PGIT, July 25, 1958. This work 
was supported in part by the U. S. Army (Signal Corps), the U.S. 
Air Force (Office of Sci. Res., Air Res. and Dev. Com.), and the 
U.S. Navy (Office of Naval Research). 

+ Faculté des Sciences de Poitiers, France; formerly with 
Res. Lab. of Electronics, Mass. Inst. Tech., Cambridge, Mass. 

t Res. Lab. of Electronics, Mass. Inst. Tech., Cambridge, Mass. 

1A. A. Sardinas and G. W. Patterson, “A necessary and suffi- 
cient condition for unique decomposition of coded messages,”’ 
1953 IRE Natronan ConventION ReEcorp, pt. 8, pp. 104-108. 

2B. Mandelbrot, “On recurrent noise limiting coding,” Proc. 
Symp. on Information Networks, New York, N. Y.; 1954. 

° B. McMillan, “Two inequalities implied by unique decipher- 
ability,’ IRE Trans. on Inrormation TuHkory, vol. IT-2, pp. 
115-116; December, 1956. 

4M. P. Schiitzenberger, “On an application of semi-group 
methods to some problems in coding,’ IRE Trans. on INFORMATION 
Tueory, vol. IT-2, pp. 47-60; September, 1956. 

®>R. 8. Marcus, “Discrete noiseless coding,’ S. M. thesis, Dept. 
Elec. Eng., M. I. T., Cambridge, Mass.; January, 1957. 


STATEMENT OF THE PROBLEM 


Let us consider an information-carrying channel with 
D symbols, d;, 7 = 1, --- , D. For any given string of 
symbols, s, we write |s|; = the number of occurrences of 
symbol d; in s, and |s| = the total number of symbols 
in s. Thus, |s| = >.; |s|;. A code word, w,, is a particular. 
s. The code-word set, Po, is a set of M code words. Sentences 
are strings of words and they form the infinite set 
P = {P,}. It is always understood that the lengths of 
the code words are bounded. Without this hypothesis, 
the conclusions are somewhat different. 

It is convenient to associate with the set {d,;} an 
arbitrary set of probabilities, p; (> Jp; l pe 
j = 1,--: , D). Then we write Pr(s) = [],; p!*'’. We may 
now define the generating function of the words, dp, (t): 


mm 


dp, (b) = > Lee (w,)t\*' = DS, ane (1) 
k i=1 
where 
“i = yp Pr (wy) 
lwrel =a 
nh, = Max 1) ee 


Similarly, we define the generating function of the sentences, 
P(t): 


(i) = iv Pr@il! = DO Axe, (2) 
where : 
A, = >, vos) Pr (8) 


|sl=n 


v(s) = number of decompositions of s into words. 

A code-word set, Po, is then uniquely decodable or, let 
us say, Just decodable (d), if v(s) = 1 for all s in P. (Of 
course, v(s) = 0, if s is not in P.) P, is said to be full (F) 
if no word can be added to P, to form a code-word set 
that is decodable. Po is said to be completable (C) if any 
string, s, can be made to fit in P by adding some suitable 
prefix and suffix. (Symbolically, we write: P, is C if) 
Vsdizand y2xsy e P.)® i 

The four theorems that will be presented show that the | 
four following statements are equivalent for decodable 
code-word sets: 

lips) Jena, 

II. P, is completable. 

Ill. gp, (1) = 1 for some particular p, set. (3). 

IV. @p, CG) = 1 forall pysets: | 


° The symbols «sy denote the string x, followed by the string s, 
followed by the string y. Here x and y may vary for different s’s. 
VY means for all; J means there exists; 3 means with the property 
that; « means belonging to. 


959 


‘THEOREMS 


heorem I: If Py is C, then ¢p,(1) = 

Method of Proof: Since the sentences are defined re- 
ursively, the A, are given by a difference equation and 
re the sums of roots to the nth power, as shown in section 
)..For Py completable, we show that the A, cannot 
ecome vanishingly small, as shown in sections 2)—4). 
ut for P, decodable, the A, cannot become larger than 
ne. Thus the root of minimum modulus, the real root, 
1ust be one. 

Proof: 


1) Av BTS, (4) 
where 7’; are the roots of ¢p,(t) = 1 


B, are constants. 
1q. (4) is true, since A, is given by 


ee (5) 
i=1 


‘he solution of the difference (5) is given by 


4=1 
vhere p, are roots of p*” — aip"" > — --- — a, = 0 
B,; are constants. 
setting T = p , we have 
Nem Glee” a 7 — =e (i ae 0 
= 1 —4,1— 4,1" =.->: — 4,0 
== dp AT). 


‘his proves (4). 

2) If Py is C, then the number of symbols in any prefix 
nd suffix that is needed to make s in P is bounded. 
More specifically, we have 


|r iene vat ese Pe 


This is obviously true, since if |z| > n,, we could break 
p x into words and a string x’ with the property that 
c'| < n,,. This x’ could serve as a suitable prefix; similarly 
or Y. 


Py ihc (7) 


a+L 


Sites ©sinen >> A, > C, > 0; forany a). (8) 


n=a 


‘o prove this, let wu; be the D“s 3 |s| = a. (See Fig. 1.) 


Let u/ be one zu;y « P 3 |x| + |y| < L. Hence, 
a <|ui|<a+ L. 


ome of the w/ may be the same but we can pick a set of 
istinct wu’, say v;, with the property that each wu; can be 
xpanded into at least one v;. Let u;,; be the set of wu, that 
an be expanded into a given 0;. 
Let 
Pr = Ere, ): 


Ujeuiy7 
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ee a L - 


Fig. 1—Abstraction from code-word tree. 


Then 
% der, (U;.;) es 


Now from each wu;,; pick the u; (call it w;) with the 
maximum Pr(u;). The maximum number of wu; in any 
u;,; 18 \v;| — a + 1. An upper bound on this number is 
(a+b) -e tl =L 41, Those yee 
Pr(u;,;). But Pr;) = Pr@) Pr(w;) Pry) 2 Pin Pri), 


DRE (a eal 


where Pmin oe min {p;}. Hence, 
a+L 
DE ADs dy Pr (0) > Pain 2 Pr (w)) 
pas Dees 
min DY 1 bit 
2 qe pes Pt Ge) eae eee 


This proves (8). 

4) Hence, lim ,.. A, > (\/(L + 1) = 
limit exists). 

5) Hence, |7,| < 1, where 
If |T,| > 1,4, ~Oasn—> o~. 

6) A, must be bounded. If A, were not bounded, then 
v(s) would be greater than one for some s because 


A, = >. o(s)Pr(s) and >> Pr(s) = 


lsl=n |s|=n 


C, > 0 Gf the 


This would mean that P, is not d, contrary to the 
hypothesis that Po is C. 

7) Hence, |\7,| > 1. Otherwise A,, would be unbounded. 

8) Since all the coefficients of ¢p,(t) are positive, dp, (t) 
is monotonic and @¢p,(t) = 1 has one real root, and no 
other root has a modulus smaller than this.’ 

QyeHence sii al ese 

10) Hence, ¢p,(1) = 1 (and this is true for all p; sets). 


Theorem IL: Vi ép.(1) = 1 and Po 1s d, then Py isi: 

Proof: If we add a word to P, to give Pé, then dp’, (1) > 1, 
and 7’, the real root of ¢p:,(t) = 1, is less than one. But 
by Theorem I, section 6), and Theorem I, section 7), 
this implies that Pj is not d. Thus P is F. 


7 The fact that the real root has the minimum modulus follows 
from Cauchy’s theorem. Cf. Morris Marden, “The Geometry of 
the Zeros of a Polynomial in a Complex Variable,’’ Mathematical 
Surveys No. III, American Mathematical Society, New NOs, IN, Woe 
1949. See especially Theorem (27.1), p. 95. 
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Theorem IIT: lf Py isd and @p,(1) = 

Poise: 

Proof: Suppose Py is not C. Then J sy 3 Vax, y «soy ¢ P. 
Since no s with sy as a prefix is in P, those strings in that 
part of the tree that from s) can be eliminated 
as possible s in P. This means that for n > 


An Sl = ErAso) 


Of those strings that do not begin with s , we can 
eliminate that fraction whose second |so| symbols are so. 


| for a given p,, then 


“orows”’ 


So ) 


Thus 
Avex (il. = PPG), tore eee eae 
Similarly, 
An <b — Pr (ale “tor, Mice sale 
Hence, A, — 0 as n — o. But for given p;, T; = 1 


and A, > C3; > 0 for some n > N for any N. Hence, we 
have a contradiction and P, 1s C. 


Theorem IV: If Py) is F, then Po is C. 

Method of Proof: Assuming that P» is not completable, 
we consider the string, wu, which cannot be completed. If 
we add wu as a word to Py, we obtain a new set, Py, which 
cannot be decodable. We then show that this implies that 
u has the same string of symbols in its beginning as at its 
end, as shown in section 14). But this leads to a contra- 
diction. 

Proof: 

1) Assume that P) is F but not C. 

2) ence, di 3° V2, yee elo} 

3) Consider Py = P, Uwand P= 4Po\ 

4) Since Py is not decodable, J v with two decompo- 
sitions in P. 

5) Choose v as a minimal doubly decomposable string 
(minimal d.d. string); that is, a string that cannot remain 
d.d. if any symbols are removed from its beginning and/or 
end. 

6) Since P, isd and P, is not, one of the decompositions 
of v must contain wu as a word. Thus v = 2,uy,, where 
Gi Uae ip 

7) Since wu is not completable in P, v ¢g P. 

8) Butve P. 

9) Hence, the second decomposition of v also contains 
Ui, OC, O = WES xe 

10) Assume that |2,| < 
designations. 

LL) los 1a), i ae 2) then ar eanclatore: 
to be d.d. either x, = 2. is d.d. or y; = y, is d.d., contrary 
to the hypothesis that v is a minimal d.d. string. 

12)s Hence; |a5) <=, aval 

13) Let us so choose the second decomposition that 
\vo| < |a,| + |ul. (See Fig. 2.) 

Otherwise, x. contains wu and must be decomposed as 
Ly = x,Uy; by the same reasoning that led to section 9). 
Thus we could have chosen to consider the first wu as the 
word wu in the second decomposition of v. 

14) Thus wu = Xylo = Uys. (See Fig. 3.) 


\v.|. If this is not so, reverse 
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Ma rch 


15) We can find (as we shall show) aw’ for which the 
equation of section 14) cannot be satisfied. Hence, the 
assumption that P, is /, but not C, which leads to this 
conclusion, is false and the theorem is proved. 

16) To find w’ we consider two cases that cover all the 


possibilities. 
Oxcibrn Sa. 
Case 2): u = aby; 0 =< b= ule ge 


We have arbitrarily called the first symbol in wu “a@’ and 


the first symbol in w which is not a, if such a vii 
exists, “6”: 
17) For case 1), let u’ = ub = a'”'d. 


Clearly, wu’ cannot satisfy 
ul = Bw = Wye; | Yo| > 0, 


since w must start with a and end with b. 

18) For case 2), let wu’ = ub!'. 

|w| < |u| is clearly impossible, for then w would have to 
start with “a” but consist only of b’s. 

But if |w| > |u|, we can write 


r+ ful 
Ww = 2,ab ; 


where 0 < |x,|;0 <r < |ul. 


Y3 
yf f+ —__ 


Xo C Yo 


Fig. 2—Grouping of symbols in the string v. 


\ u b 
CMe Eters so 
X, a pre lvl ' 
w : 
' 


X5 }+-— w a 


Fig. 4—Grouping of symbols in the string 


u 


Then, as is apparent from Fig. 4, u’ = x;w requires that 
the ug? in question occur in a position that must be a 
“b” from the fact that u’ = wy,. This contradiction 
shows that the given u’ for case 2) does not satisfy section 
(14). Thus section 15) is proved and Theorem IV, in 
turn, is proved. 


CONCLUSION 


The four theorems, taken together, show the logical 
equivalence of the four properties of the statements of 
equation 3), as is indicated in Fig. 5. 
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EG ioe 


Fig. 5—Diagram showing the relations of the four theorems. 


Sections 1)-5) of Theorem I then show that the prob- 
ities associated with a full code-word set must sum at 
st to one. Sections 6) and 7) of Theorem I show that 
$s sum must be no more than one if the code is decodable; 
itis, @p,(1) < 1 if Py isd. It can easily be shown that 
s inequality leads to the generalized Kraft* inequality 


®L. G. Kraft, “A device for quantizing, grouping and coding 
aie modulated pulses,” S. M. thesis, Dept. Elec. Eng., 
I. T. Cambridge, Mass.; 1949. 


f2 


where q, is the normalized cost of word w,. 

Our discussion shows that the equality sign holds only 
when P, is full. This inequality was obtained by Marcus’ 
by extending Mandelbrot’s proof’ for the equal-cost case. 
Mandelbrot used Shannon’s Fundamental Theorem for 
Discrete Noiseless Channels’ and pointed out that a 
similar inequality had been obtained previously by 
Szilard. McMillan® obtained a proof in the equal-cost 
case without using information-theory concepts. Note 
that the proofs of this paper are also independent of the 
Shannon theorem. 

For the equal-cost case, the normalized cost is just 
qe = n, log D, with n, = |w,|. Thus the inequality reads: 


M 
Sy De < 
k=1 


°C. E. Shannon, “A mathematical theory of communication,” 
Bell Sys. Tech. J., vol. 27, pp. 379-423; July, 1948. 


On a Property of Wiener Filters’ 


MOSHE ZAKATT 


ummary—Let Y(w,a) be the Wiener filter designed to yield 
output which is the least-square approximation to s(f + a) 
ere s(t) is the desired signal input. Let Y,;(o) be the Wiener 
sr designed to yield an output which is the least-square approxi- 
tion to some linear operation Z on the desired input signal. 
e following simple relationship has been shown to hold between 
», a) and Y,(a). If s(t) is the desired input signal and L,[s(t + a) 
he desired output, where L, is some linear operation with respect 
x, then Y,;(o) = L,[Y, a)]. 


1). Let s(t) be the signal process and _ let 

Y(w, a) be the physically realizable transfer 
ction that yields an output which is the least-mean- 
lare approximation to s(t + a). A generalization of 
s problem is to ask for the filter Y;(w) which yields the 
st-mean-square approximation to f(t), where f(t) is 
» result of some linear operation on s(t). For example, 


VE consider the well-known Wiener problem (Fig. 


* Manuscript received by the PGIT, March 24, 1958. This is 
ondensed version of Tech. Rep. T-8/ 133, written while the author 
3 at Columbia University Electronics Res. Labs., New Yo1 
Y., July 15, 1958. The research reported was sponsored by fic 
ctronics Res. Dir., AF Cambridge Res. Cntr., Air Res. and 
v. Com., under Contract AF 19(604)-1572. 

t Scientific Dept., Ministry of Defence, Israel. 


fy = we 


One i Ree 
As is well known, Y,(w) can be found by solving the 
problem for the given operation.’’” It is the purpose of 
this paper to show that once we know Y(w, a) for all real 
a, and for a given set of spectral densities G,(w), G,(w), 
G.n(w), Grs(w), we can find Y;(w) directly from Y(w, a) 
without any further references to the spectral densities 
from which Y (w, a) was derived. 

Furthermore, we can find Y;(w) from Y(w, a) by the 
following operation. Because of the linearity of the 
operation we can write 


f{Q = L,|slé + @)] (1) 
where f(t) is the desired output, s(t) is the signal input 
and L is the linear operation with respect to the variable 
a (eacine from s(t + a) to f(t). Then Y;(w) is given by 

Yi) = L.[Y, )]. (2) 


1H. S. Tsien, ‘Engineering Cybernetics,’ McGraw-Hill Book 
Co., Inc., New York, N. Y.; 1954. 

21, A. Zadeh and J. R. Ragazzini, “An extension of Wiener’s 
theory of prediction,” J. Appl. Phys., vol. 21, pp. 645-655; July, 
1950. (See ex. 1, p. 653.) 
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A similar result holds for the impulsive responses. lor 


example, let f(t) = ds(t)/dt. 
Then 
1) = ds(t + a) | Re (2) 
da [Xo Cal a6 
Therefore 
: (dY(w, 
ra) = (2a) 
da a=0 
As a second example we consider the operation of 
integration 
= | 3) de; 
then 
Lalo) Io) ce ae 
J0 
and 


nay 


Yi, = | Y, a) da. 
0 


The proof given below can easily be extended to show 
that (2) remains true for the Zadeh-Ragazzini finite 
memory filters with stationary inputs’ and for multiple- 
input filters.” 

We consider now the mean-square error of the system 
of Vig. 1. If the desired operation L on the signal is 
characterized by a transfer function Q(w) then the ex- 
pression for the mean-square system error expressed in 
the frequency domain is given by’ 


Gree 1 - -Y x }2 yr 
SUS jp (Grey =. Oi sa eae) 


seer eA MAME SEC al 1 || sn Cae 
This expression can also be written as 
See o [ yz tie. *7 p en tee hee ait 
e (t) — =| } Q) G, ie y Q Coy (3) 
Nope ena ei Ec ig eye 


We now set up the following Hilbert space interpretation 
of the optimization problem.’ The following is an ex- 
tension of the Hilbert space representation for the noiseless 
prediction problem’ to include noisy inputs. We consider 
the linear vector space of vectors of the form 


N =] 
Ss Ones 


Rey =| ; (4) 


M 


esdee: 4 


LU n=0 = 


The scalar product [K,(), K2(w)] is defined as 


3.N. Wiener, “Interpolation, Extrapolation and Smoothing of 
Stationary Time Series,’’ John Wiley and Sons, Inc., New York, 
N. Y.; 1950. 

4 Tt was suggested to us by Prof. L. Zadeh that by sacrificing 
rigor, (2) can be derived directly from the Wiener-Hopf equation. 

5 J. L. Doob, “Stochastic Processes,’’ John Wiley and Sons, 
Inc., New York, N. Y., ch. 12, sec. 5; 1953. 
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f(t)=L,[s(t+2)] 
Q (#) 


SIGNAL 


OUTPUT 


{s(t)+ n(t)} 


{n (t)} 


Fig. 1 
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Completing the space by adding to it limits in the mean 
we obtain a Hilbert space H. 

(Q(w) was defined to be the transfer function associatec 
with the desired linear operation on the signal (Fig. 1) 
Therefore’ Q(w) is either of the form of g(w) where | 


N 


O(a) kaa cae 


v=0 


(a, real) 


or Q(w) is the limit in the mean of a sequence of suck 
finite sums. The vector 


. 


therefore belongs to the Hilbert space H defined above 

Y(w) (Fig. 1) is the transfer function associated with ¢ 
lmear operation on the past of {s(t) + n(t)}. Therefore 
either Y(w) has the form 


N 
Vio)r= De Cea 


v=0 


(a, real and non-negative) (5 


or else Y(w) is a limit in the mean of such expressions. I 
other words, since the spectral density of {s(f) + n(t) 
is given by (G, + G, + G,, + G,.), either Y(w) is of ti 
form given above or else there exists a sequence Y,(w 
with all Y,,(w) of the form given above such that as n > «© 


ee 
a | Get +6n +6.) | ¥ — ¥, do 30, 
Tv —o 


but 


; 5 y* iG, 
(G, G,, —- Gen = Ga) | a | ms | 
Veale Gare 


a 


G,, ae 


Hence the set of vectors [Y, Y]” constitute a subspae 
Hot Hi. 
If we write the relation between the desired output f(i 


§ Ibid, ch. 11, sec. 9. 
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| the signal input s(¢) in the form given by (1) then 
Q(w) = L,[e'**] 


are Lfe'“*]is of the form of the finite sum ye Oe 
v limit in the mean of such sums. Therefore (2) can be 
tten as 


bl | a ee ; 
0 L 0 
1 we have to prove that L, and P commute. Let 
baler bal ee 
Ve OF yee LO 
n 
° 1 Aa 8| a a ACS ie 
aY, + BYs 0 4 


erefore L, and P commute when L, is of the form of 
inite sum > a,e’“’*. We will prove that L, and P 
nmute also when L, is a limit in the mean of such 
ite sums by proving the following lemma: 

f {x,} is a sequence of vectors in a Hilbert space H con- 
ging to a vector x, then the sequence {Px,} where 
is a projection from H on a subspace H, is also con- 
gent. Moreover, if € = lim,.. {Pz,} then & = Pz. 
of: If x, — x (in the mean), then given any « > 0 
re exists an m such that || xz, — &,, || < e for all 


n,m > nm. For every vector y in H we have 


UO eee eR) 
therefore, 
|| Px, — Px» || = || P@, — 


ra) ale Sig epee a oes 


Because of the completeness of H, it follows that there 
exists a vector € in H, such that Px, — & Therefore, for 
e > Oandn > m we have 


EP aaa Peete 
and 
||é — Pa, 


| Ge: 
Hence, by the triangle inequality, 


OPTS |e 


andePe = =. 

In the proof given above we have implied that G,(w) 
and G,,(w) be integrable (— ~, o). This requirement is 
not essential. What is important is that G,(w) and G,(w) 
\Q(w)|* be integrable (— ©, o). If this requirement is met 
and G,,(w) is not integrable (— ~, «), then we can replace 
the elements ae'*” appearing in (4) and (5) by other 
elements such that the norms ||Q|| and||Y|| will be finite 
and @ and Y will correspond respectively to linear opera- 
tions on the signal and linear operations on the past of 
the signal plus noise. 


Machine Recognition of Hand-Sent Morse Code’ 


BERNARD GOLD{ 


ummary—A transistorized special purpose digital computer 
ed MAUDE (Morse AUtomatic DEcoder) has been designed, 
it and analyzed. This computer decodes a hand-sent Morse 
ssage, which is printed on a teletypewriter. 

IAUDE’s decisions take place at a number of different levels 
its “knowledge” is not only that of relative durations of dots 
dashes, but also of the Morse code and even of certain elemen- 
’ properties of language. 

fAUDE has successfully decoded between 90 per cent and 
per cent of 184 operators. A successful decoding is one which 
ilts in a text which can be easily read by a man who knows 
language. 

t is felt that MAUDE can be a practical piece of equipment 
a site with heavy traffic. Its performance will be inferior to 
t of a man until more sophisticated language rules, using at 
st a word vocabulary, are included. 


* Manuscript received by the PGIT, November 11, 1958. The 
arch in this document was supported jointly by the Army, 
vy, and Air Force under contract with the Massachusetts 
titute of Technology. 

t Lincoln Lab., M. I. T., Lexington, Mass. 


I. IntRopucTION 


HE DESIGN of the Morse automatic decoder is 
A ees primarily on observations of actual hand- 

sent Morse code messages. It will not, therefore, 
be explained on other than pragmatic grounds or its 
history traced. Rather, the rules governing MAUDE shall 
be presented briefly with a subsequent interpretation of 
the voluminous data that was a necessary product of 
this project. The Appendix presents some mathematical 
analyses pertaining directly to MAUDE. Two companion 
papers are in preparation—one describes in detail the 
engineering design; the other is a comprehensive mathe- 
matical treatment of a statistical problem which arose 
from the MAUDE project. 


Il. DEFINITION 


Morse code is a rudimentary encoding scheme for 
language in which each character (letter of the alphabet, 
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number or punctuation sign) is represented by a unique 
sequence of marks and spaces. The international Morse 
code is given in Fig. 1. 

The accepted definition of an ideal, or machine, Morse 
code is one for which the five elements have the following 
relative durations: marks—dot = 1 and dash = 3; spaces— 
symbol space = 1, character space = 3 and word space = 
7. Symbol spaces separate marks within a character, 
character spaces separate characters within words, and 
word spaces separate words. Symbol spaces shall be 
referred to as short spaces, and character and word spaces 
as long spaces. 

The three standard instruments for generating a Morse 
signal are 1) the simple hand key, 2) the semiautomatic 
key, or “bug” and 3) the automatic Morse machine. The 
hand key because of its simplicity and low cost is most 
often used. Marks (dots and dashes) are produced by 
depressing the key; spaces are produced by lifting the key. 
The durations of all marks and spaces are controlled 
directly by the sender. A “bug” is more difficult to 
control and is generally used by more experienced oper- 
ators. This key has two “on” positions, one of which 
causes a machinelike sequence of dots to be generated. 
The dashes are produced manually from the other “on” 
position. The transmission and reception of completely 
automatic Morse are a routine matter which is not of 
concern here. 


II]. ENcopiInG AND LANGUAGE 


At this point the thought implicit in the first sentence 
of Section II should be expanded. Morse code is itself not a 
language but a way of representing or coding a given 
language, such as English or German. It is analogous to 
handwriting in that there is a symbol for each character in 
an alphabet. In fact, some intuitive feelings about the 
Morse code problem may be developed by pursuing the 
handwriting analogy. Some people write very clearly so 
that anyone can read their writing and, further, even those 
who cannot understand the language of the clear writer 
may still be able to understand and reproduce each symbol 
(letter of the alphabet) that was written. It is, however, 
very helpful to know the language being written; in fact, it 
is usually much more difficult to read handwriting in a 
foreign language, even if this language is somewhat 
familiar, than to read one’s native language in the same 
handwriting. Thus we see that, for handwriting, knowledge 
of the code is sometimes (in the case of the clear writer) 
sufficient to discern each intended symbol; in many cases, 
a higher knowledge, 7.e., of the language, is necessary 
to do so. Note that understanding of the meaning is not 
being discussed, but merely the identification, symbol for 
symbol, of what was written. This is all that is attempted 
in MAUDE. 

A typewriter (or a Teletype machine) performs a 
noiseless coding; the process is exactly defined and perfectly 
predictable. For these codes, it is clear that only the en- 
coding procedure need be known in order to be able to 
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Fig. 1—The international Morse code. 


decode any message perfectly. Handwriting, speech or 
hand-keyed Morse code, on the other hand, are examples 
of noisy coding. Of these three examples, Morse code 
appears to be by far the easiest to decode, and yet even 
here it is not out of place to inquire to what extent a 
decoder (man or machine) needs to use only his knowledge 
of the code and to what extent he needs to use his knowl- 
edge of the language being encoded. | 

These questions will be taken up again later, after a 
discussion of the design and description of the operation. 
of MAUDE. | 


IV. Tue Frxep RuiEs 


The crux of the difficulties encountered in designing 
MAUDE is the variability among individual senders. 
The main problem is to find a set of rules which will work 
for nearly all operators. 

The fixed rules of MAUDE are such rules. Through 
their use, a Morse message may be partially decoded. 
They are rules based on the properties of the encoding 
scheme (7.e., Morse code) and, in a rudimentary way, on 
the structure of language. 

At this point, it was noted that a man, who knows 
Morse code, will never remember a sample of data as 
accurately as a machine; yet he decodes a Morse message 
well enough to make any machine thus far built turn 
green with envy. Why? The answer can only be that to a 
very great extent he knows what to expect because he 
knows Morse code and the language being sent. 

To illustrate, the first two letters of “jumped,” taken 
from an actual Morse code message, is considered (See 
Fig. 2.) The numbers refer to the relative duration of 
the marks and spaces. It is clear that on a purely statistical 
basis, the number 15 would correspond to a short space. 
However, the result would then be meaningless as Morse 


59 
de. In order to make it meaningful, there must be at 
ast one additional long space between 68 and 33. Since 
is the longest of all other spaces, it then becomes 
ghly probable that 15 belongs to the set of long spaces, 
hich, in fact, it does. 


|< sol El prey ae me ka pe Neen 
| SPACES 
33 


g. 2—Morse code for J U. Top numbers are mark durations; 
bottom numbers are space durations. Scale: 1 unit = 5 msec. 


Now the fixed encoding and language rules can be 

ated. 

A) The longest of six successive spaces is almost always 
a long space. For brevity, this is called the rule Lg. 

B) The shortest of six successive spaces is almost always 
a short space. This is the Rule S,. 

C) The shortest of three successive spaces from the set 
of spaces defined by ZL, is usually a character space. 
This is the rule S,. 

D) The longest of three, four or five successive spaces 
is almost always a long space if the succession of 
marks which they separate do not constitute a 
character of the Morse alphabet. (For example, four 
successive dashes do not constitute a single character. 
Thus the longest of the internal spaces is probably 
a long space.) This is the rule L,, where k = 3, 4 
or 5. This rule will also be referred to as the alphabet 
test. 

E) The shortest of six successive marks is usually a dot. 
This is M,. 

All of these rules are applied to sliding intervals of 

arks (or spaces). In this way, the greatest number of 
ymbols is classified. 

Before a discussion of these rules, it was noted that Lg, 
, and S, can be used to categorize some of the spaces 
\to two sets, short and long; S; can help separate some of 
1ese long spaces into character and word spaces and M, 
elps distinguish dots from dashes. A step has therefore 
een made towards the required ternary decision on spaces 
nd binary decision on marks. 

The rules L, and L, are based exclusively on the nature 
f the Morse alphabet. As seen from Fig. 1, this alphabet 
mtains all possible combinations of 1, 2 and 3 mark 
1aracters, but appreciably less than half of the 4, 5 and 6 
ark characters and no characters of more than 6 marks.’ 
hus, at least one of any 6 successive spaces and at least 
ne out of certain sets of 3, 4 or 5 successive spaces is a 
mg space; this datum has convinced us that these long 
aces are very rarely of lesser duration than any of the 
eighboring short spaces. 

The rule S, is based on the observation that five or more 


1 Tf the error sign is called a character, it is the one exception 
. this statement. Section VI describes the special treatment needed 
r error signs. 
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successive one-symbol characters (# or 7’) rarely occur in 
English. For random letter cipher text, the probability 
of occurrence is (1/13)’. The data convince us that the 
true symbol space will not be of greater duration than 
nearby long spaces. 

The rule S, is based on the rare occurrence of two 
successive words with no more than five internal spaces 
since in this event the rule L, may pick out three successive 
word spaces. The rule M, works well except when six 
successive dashes are present. 

All the rules discussed are based on the structure of the 
encoding and the language. Another rule, based on 
operator sending characteristics, resulted from the dis- 
covery by Smith-Vaniz’ that spaces following dashes tend 
to be shorter than spaces following dots. He suggested 
that a percentage of the previous marks be added to all 
spaces, so that new space durations S’ = S + aM, 
where S and WM are the true durations of the space and 
the preceding mark, were used. In this connection, it was 
noted that characters ending in a dash (A, M, O, etc.) 
tend to be run together with the following character. In 
fact, the rules L, and L, have, because of this, failed fre- 
quently enough to cause concern. The transformation to 
S’ improves this situation appreciably. 

The constant a, as might be expected, is really a funec- 
tion of the operator. However, a = 0.18 has been found 
to result in a fairly universal improvement for hand-sent 
Morse. The transformation is more of a hindrance than 
a help for “bug” sending, yet not harmful enough to 
deterioriate noticeably the performance of MAUDE. 

It is clear from this discussion that these rules are not 
absolute. However, their application has proved very 
helpful. 


V. Tur THRESHOLD TESTS 


Three thresholds, 7’, T's and 7’; are used (see Fig. 3) 
to classify the remaining marks and spaces (those not 
picked by the fixed rules). These thresholds will fluctuate 
during a given message as well as from message to message 
and for different operators. 0, 0; and @¢ are running 
averages of those marks and spaces “tagged” by the 
language rules and thus the 6’s should accurately mirror 
the properties (such as word speed) of the particular 
message. Ay, Ag and Ag are empirically determined 
constants,’ whose values, ideally should be such that the 
T’s are always at their optimum setting. 

The situation is described statistically in Fig. 4. Let 
f, and g, be the probability density functions for the 
short and long spaces. Decisions are now made by the 
rules S, and L, and those spaces remaining are shown by 


2 Private communication. 

3 In Fig. 2, all variables are actually logarithms of time durations. 
This is accomplished in the equipment by transforming the durations 
of all marks and spaces so that 2 = logig/is y where y is the original 
duration. The A’s in Fig. 2 thus correspond to ratios, which, it is 
felt, contain less inherent fluctuation than differences. Also, the 
dynamic range is increased. 
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Fig. 3—Thresholds used in mark and space decision. 
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Fig. 4—Density functions f; and g; for all short and long spaces, 
respectively. fz and g2 indicate the distribution of spaces remaining 
unidentified after the operation of rules Ls and Ss. 


fe and g.. A threshold 7's which adequately separates 
f. and g, must now be defined. 

Empirically, the following rule has been established: 

Let 6s(n) be defined by, 

Os(n) = Os(n — 1) + ; [xs(n) — As(n — 1)] (1) 
with the initial condition @s;(1) = ws(1). as(n) is the nth 
space picked by Ss. @s(n) can thus be considered as a 
weighted average of the first n spaces “tagged” by S,. In 
fact, (1) can be solved by iteration to give 

dsm) = ase +1 — 8) Deas" — @) 

where € = 1 — 1/8. The value 6 = 4 has been used. 

A simple calculation shows that 9, the mean of 6s(n), 
is always the same as @, the mean of all the x s(z). Thus 
6s(n) is the sample mean of the density function defined 
by fo. 

As ascertained from data, 

1) T's can be formed by adding a fixed value of As to 

6s, with generally good results. 

2) If As is chosen for each operator and then fixed for 

the entire message, a worthwhile improvement over 
1) is obtained. A detailed analysis of data based on 
both 1) and 2) is given in Section IX. 

Similar arguments hold for the thresholds 7’, and T'y. 
Since the rules 1/7, and S; are not as reliable as S, and Lg, 
the former are used only to generate 6,, and 6,;. The 
thresholds 7’), and 7, are then used to make all decisions. 
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VI. Tur Error SIGN 


When an operator makes an error and realizes it, he 
sends an error sign. This most commonly consists of a 
sequence of 6 to 20 dots. However, practices vary— 
sometimes the characters IMI are sent, other times simply 
II is sent. 

If the fixed rules were applied to the common error sign, 
the values of the thresholds would be seriously affected. 
Since error signs occur fairly often, it was necessary to 
have an error sign detector whose output (when an error 


sign occurred) could inhibit the remainder of our system. 


Upon observation of the properties of actual error signs, 
the following criterion was empirically established: if five 
or more successive logarithmic differences between 
adjacent marks and between adjacent spaces are each less 
than a fixed number, an error sign is present. Recent 
experiments indicate that a fixed logarithmic difference 
of 10 (corresponding to a ratio 2:1) does the job. 


VII. SEQUENCE OF OPERATIONS 


The laboratory model of MAUDE is a special purpose 
digital computer using transistorized flip-flops as the 
basic storage unit. A block diagram is shown in Fig. 5. 
In this section the sequence of operations shall be de- 


scribed briefly. A fuller description is contained in a com- — 


panion paper. 
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Fig. 5—Block diagram of MAUDE. 


The logarithms of the durations of alternate marks and 
spaces are stored digitally in the logger. The counts 


representing these logarithms are alternately passed — 


through the mark and space storage registers. 

Rules Lo, Ss and MW, are first applied. The marks picked 
by M, are used to adjust a 6,, counter as indicated by (2). 
In the present model, all mark decisions are made by 
comparing each mark with 6, + Ay, the latter being 
fixed. Decisions between short and long spaces are made 
by applying L, and S, followed by a threshold test. S, 
is performed in supplementary storage. 

Fig. 6 shows in detail how parts of two actual messages 
are decoded by MAUDE. The numbers on the marks line 
are the logs of the actual durations of marks, while the 
space numbers are the results of the transformation 
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Fig. 6—Detailed operation of MAUDE on actual message. 


s’ = S + kM, described in Section V. By following the 
‘heck marks, the reader can see which marks and spaces 
vere picked by the various rules and can mentally compute 
he thresholds. 

The alphabet test successfully separated O from § in 
»ostulates because — — — --- does not exist in our Morse 
ulphabet. However, it failed to separate T from U because 
¥ (TU run together) is in the Morse alphabet. This 
nistake could not be rectified on a letter basis, but would 
‘equire more complicated contextual rules. 

It should be noted that the man who sent the first 
nessage tends to run letters together more often than 
nost operators, yet he is by no means extraordinary, and 
wn experienced receiving operator would not have much 
rouble decoding this message. It might also be mentioned 
hat other operators do the opposite, 7.e., tend to split 
‘yharacters. 

The second operator was decoded perfectly by MAUDE. 
[his is fairly typical for about half of the operators that 
vere decoded by MAUDE. 


VIII. CLASSIFICATION OF ERRORS 


The types of errors that either a man or a machine 
vould make in decoding a Morse message depend on the 
‘haracteristics peculiar to the sender. A classification of 
he errors encountered in MAUDE translations will now 
ye given in the hope that it will give a fairly inclusive 
yicture of the errors to be expected. 

1) Failure of the rule Le. Examples: OF — TGR, 
IF — 8N, JU — EMNA, OP — TGG. In these cases, the 
ender paused for less time after the third dash in O (or J) 
han after other marks in the sequence. Thus Le picked 


the wrong space. L, did not fail often enough to make 
revision of the rule necessary. It was noted, however, that 
most of the failures involved the letter ‘O.” 

2) Failure of the alphabet test. Examples: OR — MC, 
OU — MX, OS — MB. The cause of failure is exactly 
the same as for L,. The alphabet test was applied, as it 
should have been, but the wrong space was the longest. 

3) Two characters run together. Examples: TH — 6, 
MA — Q, Al > L, AN > P, ON — 9, TW — Y. In 
these cases, the letter space, e.g., between 7 and H, was 
too short. A decision could be made only on a threshold 
basis since the alphabet test could not yield a unique 
decision. 

4) Split characters. Examples: A — HT, U — EA, 
H > II, J > £O,0 — MT, L— AI. This is the inverse 
of 3). The sender pauses too long during a character. 
Again, only a threshold decision basis is possible. 

5) Simple mark errors. Examples: A — M, F — H, 
J — P,G— OO, O > G. These are caused by incorrect 
threshold decisions on a dot or dash in a character. 

6) Complex mark errors. Examples: F — UT, P > MN, 
@ — MM, Y — OT. In these cases, a mark error (based 
on the threshold) produces an impossible sequence of dots 
and dashes. This automatically sets the alphabet test to 
work. In the present model of MAUDE the alphabet test 
can only change short spaces to long spaces. Hence, a 
mark error generates a space error. 

It is important to note that all of the above errors were 
caused by fluctuations in the durations of marks or spaces. 
Thus, it is reasonable to expect that many of the wrong 
decisions which caused these errors were close decisions, 
e.g., a slight change in the threshold value may have 
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reversed a decision. For the data analyzed, about half of 
the total errors were of the types 1) through 6). 

7) Extra dot (simple case). Examples: D — B, Kk — TJ, 
M—-W,I-S8S,O-—-Q, 8S — H, U — V. The sender 
occasionally sends an extra dot, through carelessness or 
fatigue. This happens more often with “bug’’-sent 
messages. It can also result from “key bounce” on hand 
sent or bug-sent Morse. 

8) Extra dot (complex cases). Extra dots most often 
occur when numbers are sent. This triggers the alphabet 
test so that, e.g., 6 — DI. 

9) Missing dot (simple). Examples: V — U, B — D, 
D-N,F->R,X->K,J—->0, LOR. 

10) Missing dot (complex). Examples: TNS 
D — TE, Q — MT. In these cases a longer space is 
generated where the dot was left out. 

11) Extra space. Examples: C — 6, Q — 7. A dash was 
broken up into two dots by the insertion of a space during 
the dash. 

12) Missing space. Examples: A > 7, B— k, O—> M, 
H — U, U— M. Here, either two dots are run together 
to make a dash, or a dot-dash creates a single dash. 

13) Complex errors. About 10 per cent of the garbles 
analyzed could not be attributed to any simple errors, 
eg., THE > XT, TH > KB, AC Y, JUMP > WXTY. 

Errors of the type 7) through 13) can be caused by 
operator sending lapses and also by noisy radio reception, 
e.g., fading, atmospheric noise, adjacent channel inter- 
ference, etc. It is more difficult to correct such errors 
because the origin of the garble cannot be easily located. 


C= 


IX. Anatysis or MAUDE PERFORMANCE 


Since operator variability causes the main difficulty, 
any judgment on a Morse decoder must be based on data 
from many different operators. Furthermore, there must 
be some idea as to how the error rate affects the ability 
of the human observer to understand the messages. For 
example, if MAUDE decoded all messages with no more 
than 2 per cent of the characters garbled, there would not 
be too much point in improving MAUDE to make the 
maximum error rate | per cent since it has been ascertained 
that people can correct texts containing 2 per cent and 1 
per cent error with almost equal ease.* 

It is felt that the effectiveness of a machine such as 
MAUDE depends on the percentage of received messages 
that it can adequately decode. The word “adequately” 
pinpomts the vagueness of this statement; this is inter- 
preted to mean “‘capable of reconstruction in a reasonable 
length of time;’ even here the problem becomes too 
subjective. Miller’s paper and the tests run by Van Hoosen 
at Lincoln Laboratory” give some insight into this problem 
and the interested reader may refer to these works. 


4 Referred to, of course, is texts of a reallanguage that the observer 
knows. See G. Miller and E. Friedman, ‘“The reconstruction of 
mutilated English texts,’ Information and Control, vol. 1, Sep- 
tember, 1957. 

5 Van Hoosen, Lincoln Lab., M. I. T. Lexington, Mass., Quart. 
Prog. Rep.; July 15, 1958. (Not generally available.) 
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Table I shows the total number of messages with less” 
than a given percentage of character errors out of 184_ 
messages sent by 53 different operators (containing about 
45,000 characters). Table II shows a similar compilation 
for space errors, defined as a letter space mistaken for a | 
word space or vice versa. Table III shows error rates for 12_ 
operators for whom English text, number cipher text and_ 
letter cipher text were obtained. There was no significant | 
difference in the quality of the different types of texts as_ 


translated by MAUDE. | 
TABLE I 


NumBeEr or Messaces, NV, Havine Less THAN THE GIVEN 
Prr Cent CHARACTER HRRORS Y 


Per Cent 
Character Errors N 


>) 
bo 
Or 
pe 
ww 


or 
— 
— 
i 


CONOOEWNRE KOO 
= 
iS 
or 


10 169 
over 10 184 


TABLE II 


NuMBER oF Messaces, V, Havine Luss THAN THE GIVEN 
Prr Cent Space ERRorS 


Per Cent 

Space Errors N 
1 5 

2 18 

3 34 

4 53 

5 71 

10 109 

15 123 

20 124 

TABLE III 


CHARACTER ERROR RatEs, BY OPERATOR, FOR THREE TYPES 
or Messace. THe ERRor Rate 1s Grvpn IN THE Form A/B, 
Wuere A is THE NUMBER OF CHARACTER HRRors (ALL TypEs) 
AND B tHE Toran NUMBER OF CHARACTERS SENT. 


Operator No. English Letter Cipher | Number Cipher 
1 4/655 7/1068 9/687 
2 5/459 4/678 1/475 
3 8/655 8/678 19/687 
4 8/459 29/913 10/475 
5 7/655 2/865 3/687 
6 20/655 10/703 5/475 
a 7/655 6/1068 7/687 
8 2/655 5/703 0/687 
9 5/449 12/858 10/687 
10 1/253 7/1108 1/687 
iil 1/449 6/1108 3/502 
12 4/459 10/1023 3/802) 
Composite 72/6438 = 106/10772 = 71/7088 = 
1.12 per cent | 0.99 per cent 1 per cent 
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Fig. 7—Laboratory model of MAUDE. 


For purposes of discussion, assume that a message 
ving less than 6 per cent character errors is acceptable.° 
1 this basis, note from Table I that 30 out of 184 messages 
e not acceptable. Now, Table I was based on a fixed A 
sign for MAUDE, in which Ay, As and Ag were chosen 
‘analyzing about 10 of the messages and picking the best 
lues for them. All 184 messages were then run off keep- 
7 all the A’s fixed. 

It is clear from Table I that a rather high percentage of 
e total messages was badly garbled. In order to investi- 
te this more carefully, a simulation program on the IBM 
4 was run first in which the MAUDE program was 
neralized so that it could choose its own value of A for 
ch 100 samples of marks and spaces. In order to make a 
rect comparison between the fixed and variable A 
hemes, errors only of the types 1) through 6) were 
unted and, on that basis, another table of cumulative 
stributions was constructed. (The errors 7) through 12) 
e completely unaffected by the type of variation on A, so 
ey were not included in the comparison. They are listed 
the third column of Table IV and will be discussed in a 
mment.) Irom the first two columns of Table IV, a 
ticeable difference between the fixed and variable A 


6 Tf willing to use such a crude criterion, then 6 per cent is the 
‘+ guess based on Van Hoosen’s data. The combined effect of 
wacter and space errors has not yet been checked, but the latter 
ms appreciably less significant. 


TABLE IV 


Numeer or Messages, NV, Havine Luss THan THE GIVEN 
Per CENT CHARACTER HRRORS 


Errors 1) through 6) Errors 7) through 12) 

Per cent Fixed A Variable A | Incorrect Number of Marks 
0.25 18 84 72 
0.5 87 104 79 
0.75 110 129 107 
1 120 138 122 
L& 132 147 137 
2 145 154 150 
3 158 164 162 
4 162 170 171 
5 164 175 175 
6 164 ilaea 178 
7 167 179 180 
8 169 180 181 
9 169 181 181 
10 170 181 181 
Over 10 184 184 184 


schemes can be seen. For example, using the 6 per cent 
criterion, twenty messages are unacceptable for fixed 
A’s, only seven for variable A’s. 

From column 3 of Table IV, note that messages in 
which only the type of errors 7) through 12) were counted 
caused about the same number of unacceptable messages 
as was caused by the variable A program. No way is 
known to correct these errors without the use of more 
complex contextual rules involving knowledge of word 
structure. Furthermore, even if all the errors of column 2 
were eliminated and only those of column 3 remained, 
there would still be a significant difference in performance 
between a man and MAUDE. The conclusion is in- 
escapable, therefore, that for the automatic reception of a 
language encoded by even a simple process like Morse 
code, a machine must have some knowledge of the 
language if it is to approximate the performance of a man. 
In particular, MAUDE must, to some extent, know 
English if it is to decode English sent by Morse code. 

The reader will note a return to the questions raised in 
Section III. If allowed to generalize somewhat from the 
experience with MAUDE, it would be concluded that 
efforts to decode by machine “noisily” encoded language 
symbol by symbol, e.g., speech recognizers using only the 
waveform of the speech, are bound to be severely limited. 
It must be recognized that better decoding machines 
depend not only on ingenious decoding schemes, but also 
on more use of the structure of the language and greater 
understanding of human decoding processes. 


APPENDIX 


A fairly complete mathematical theory of the rules 
L., S, and M, has been worked out and a separate paper 
has been submitted for publication. This Appendix is 
concerned with the part of the theory applied most directly 
to MAUDE which, in turn, is based primarily on a general- 
ization of a theorem due to Maximon.’ 


7L. C. Maximon, ‘Some Properties of Infinite Sequences of 
Independent Random Variables,’’ Lincoln Lab., M.I.T., Lexington, 
Mass., Group Rep. 34-53, 1956. (Not generally available. ) 
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Theorem: Let y, be the mth term (m > k) in any 


sequence Y of independent random variables. Let Y be 


operated on by the maximum rule of & (an integer > 2) 
to produce a subsequence V. The probability P(y,,) that 
Ym, beim V is 
k k-1 

PWn) = Qe p(B) — Le p(RRis) (3) 
where R; is the set of numbers y;, Y;+1, a es eae 
R;R;., 18 the set of numbers ¥;, Yi11 °°: , Yirr, DUR;) 18 
the probability that y, is the largest number in R,, 
p(R,R;,,) is the probability that y,, is the largest number 
in R,R,;,,. Note that it is identical for both minimum and 
maximum rules.” 

To apply the theorem to a Morse signal, probabilities g 
and 1 — q are assigned to the occurrence of short and long 
spaces (or dots and dashes). It can be assumed that a long 
space is always longer than a short space in a given set 
R;. Then, the probability p(R;) that a given long space is 
the longest long space in the set R; can be shown to be 


pey= pe Ueda ta» @ 


and 


“MAUDE 
Lexington, 


§ This theorem is proved in Hisenstadt, Gold, et al., 
(Morse Automatic Dezoder),’’ Lincoln Lab., M.I.T., 
Mass., Group Rep. 34-57; 1957 
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k+l pe me ot jo i 
Mini) OD (eres (5 


7=1 ] 
Using (3) and noting that p(R;) is invariant with 7, 
PY;) =k (ee (6 


Iq. (6) is tabulated in Table V. 


TABLE V 

TABULATION OF (6). 
k Gi=3 AG ee 
2 0.91667 0.9630 0.9792 
3 0.8125 0.9074 0.9453 
4 0.7125 0.8444 0.9039 
i) 0.6250 0.7888 0.8594 
6 0.5513 OF Malai 0.8113 
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: The Morse Distribution 


M. FREIMER}, B. GOLD anp A. L. TRITTERT 


‘ummary—A problem which arose during research involved in 
igning a machine to translate hand-keyed Morse code into 
ted text may be stated as follows: Let X = {x;:i=1,2,---,n} 
a sequence of independent random variables all of which have 
same distribution. Assume that the probability that x; = x,, 
E j, is zero. Let k be a positive integer < n, and consider all 
sequences X;, Xizi, °°* » Xi+x-1 Of X consisting of k consecutive 
iables. Let us distinguish, with a check (\/), the largest member 
-ach such subsequence. We have studied, and partially tabulated, 
1), the probability that exactly r members of the sequence X 
not checked. This paper contains most of the pertinent results. 


INTRODUCTION 


ORSE code (as sent by hand) is one of the 
simplest of aural languages, yet it has features 
common to all spoken languages. At the sug- 

stion of O. G. Selfridge, we have studied ways of building 
machine which can recognize Morse code. We have 
mulated certain rules, both linguistic and statistical, 
ich have proved very valuable for our purpose. Analysis 
these rules led us to investigate a discrete probability 
tribution, apparently never before studied, which we 
ull call the Morse distribution. In this paper we present 
‘tain mathematical properties of the Morse distribution, 
uch is tabulated in the Appendix. 


Lineuistic RuLES 


Morse code is a sequence of alternating marks and 
aces. Marks are either dots or dashes. For our purposes 
wees are categorized into two groups: a) short spaces, 
ich separate two marks of the same character—letter, 
mber or punctuation mark—and b) long spaces, of 
wer duration, which separate adjacent characters. 
No character in the Morse alphabet contains more than 
narks (and thus, 5 short spaces). We would therefore 
yect, and experience has proven, the following rule to 
true: 


le 1—The largest of any group of 6 successive spaces 
(almost surely) a long space. 


The second rule follows from an elementary property 
the English language (shared, it seems, by others), 7.e., 
it a sequence of 5 successive letters, each of which is 
or 7’, is very improbable.’ The Morse Code for FH is a 
gle dot; for 7’, a single dash; any other character must 
tain at least two marks and therefore at least one short 


‘ Manuscript received by the PGIT, January 26, 1959. 

Lincoln Lab., M.1.T., Lexington, Mass. The work reported 
» was performed at the Linclon Laboratory, Massachusetts 
titute of Technology, with the joint support of the Army, Navy 
| Air Force, under contract. 

1 But not impossible, e.g., “sweet teeth.” 


space. We thus establish the second rule: 


Rule 2—The smallest of any group of 6 successive spaces 
is (almost surely) a short space. 


One of the primary functions of our machine is to 
select the largest and smallest of each sequence of 6 
successive spaces and categorize those selected according 
to the above rules. One of the questions for which we seek 
an answer is, “What are the statistical properties of the 
number of spaces selected by these rules?”’ We have tried 
to answer this question both experimentally and theo- 
retically; in this paper we present the results of the 
theoretical investigation. 


Tse Morse DistrisutTion 


We define a discrete probability distribution, which we 
have called the Morse distribution, as follows. 

Let X = {2;7¢ = 1,.2,%---, 1} be alsequence ofan. 
dependent random variables, all having the same con- 
tinuous probability distribution (x). For any positive 
integer k < n, define a new sequence Z = {2;:7 = 1, 
2, -+-,n} = L,(X) such that z, = 1 if a; is the largest: 
of any set of k consecutive variables of X, and z; = 0 
otherwise.’ An example, for k = 3, is shown in Fig. 1. 
In this example x, is the largest member of the sets S, 
and S;; and x; is the largest member of the set S3. 


Fig. 1—Example of the transformation Z = L;,(X). 


Letting r = n — >."_, 2;, we define the probability 
of r, for fixed n and k, as pk(r). This is the general term 
of the Morse distribution. Physically, r represents the 
number of variables (in X) not picked out of a sequence 
of length n, by the operation L, defined above or, in other 
words, the number of zeros in Z. 


2 The problem is the same whether the largest or smallest of k 
consecutive variables is to be chosen. For convenience, therefore, 
we shall always consider only the largest. 

3 For k = 1| this is a trivial procedure, since each z; = 1. Hence 
we shall only consider k > 2. 
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Since we have assumed A(vc) to be continuous, the 
probability of x; = a; for any pair 72 ¥ 7 is zero. If we 
ignore this null event then in any sequence X the variables 
, can be ordered (by magnitude) in a unique way. We 
may therefore replace X by a sequence Y, the terms of 
which are the integers | through n in the naturally corre- 
If we apply L, to Y we get the same 


xv 


sponding order. 
sequence Z. 
We can now compute 


A 


Pn (r) = Inn) 


n! » 
where f<(r) is the number of permutations of the n integers 
which are carried by L, into those sequences, Z contain- 
ing exactly r zeros. We thus see that the distribution 
function A(x) plays no part in computing the probabilities 
Dt) 

We shall omit the superscript k when this causes no 
confusion. 


Recursion ForRMULAS FoR f,(7) AND 7,(r)* 


Let us suppose that the largest number, 7.e., n, 1s the 
2 + 1 term in Y. Then, the part of Z formed from the 7 
numbers to the left of n is zndependent of the part formed 
by the n — 7 — 1 numbers to the right of n. Furthermore, 
these first 7 places of Z are the same as those that would be 
obtained by applying L, to the sequence of total length 
7 that is identical with the first 7 places of Y, and similarly 
for the last nm — 7 — 1 places of Z. Finally, there are (";’) 
ways of choosing the numbers in the first 2 places from 
1,2,---,n— 1. From these considerations, we derive the 
following recursion formulas: 


1LO= LET NOtene-9, nz k@ 
=0 7= C2) 
Dlr) = — 2 2D Pi(j)Pn-iil? — J), n > k(b) 


For n < k, we adopt the convention that nothing is 
picked; hence, all terms of Z are zero and 


ay eh = {0, NFP 
a 


This convention is necessary in order that (2) be correct. 

Eqs. (2) make it possible to compute p,(7) numerically 
for successively larger values of n. A much simpler form 
than (2) can be obtained by defining generating functions 


(3) 


WE = Io 


4The recursion formulas (2) and the differential equation (9) 
were first derived by T. Austin, R. Fagen, T. Lehrer, and W. Penney, 
“The distribution of the number of locally maximal elements in a 
random sample from a continuous distribution,’ Ann. Math. 
vol. 28, secs. 1-3, pp. 786-790; 1957. Their work, motivated by the 
formulation of the problem, contributed to the development of this 
analysis. 
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ay) = Do pay = 


which satisfy the equations 


1 n-1 
a{y) = — > a 


To prove (5), we multiply both sides of 2(b) by y’ and 
sum over r from 0 to ©, then interchange the order of thal 
r and 7 summations on ine right-hand side. 

From (3) and (4), we obtain 


NO i aul | (5) 


>= 


ys PANY’ 5 . 
| 
| 


OQ) =~ Gis (6) 


In the Appendix, we have tabulated p*(r), obtained 
from (4) and (5), for k= 2,038)475"6, andiaanias (For. 
larger n, computation becomes increasingly laborious.) 
In addition, we have derived the following explicit eX 
pressions for a,(y), Qi+i(y), and a@,,2(y) for all k: : 


aly) = y* é 
2y"* + (ke — Dy" 

S4y't yt 
Axs2(Y)-= (k + 1)(k + 2) (c) 


DIFFERENTIAL EQUATION FOR THE 
GENERATING FUNCTION F(a, y) 


We can derive a simple recursion formula for a:(y) by 
defining the generating function 


E Y patriaty = 


If we multiply both sides of 2(b) by x"y’ and perform 
a series of manipulations, 


Y a,(y)x". (8) 


Ken wes | 
r= 


we derive the differential - 


equation 
dk S : 
at l-y Lot Dey = (Fe, yr OF 


with boundary condition F(0, y) = 1 for all y. 
For k = 2, the solution of (9) is 


Ray) = 
| — e 2 
= w{l ah Q(e?*"z - ete? +. Cue ate ise :)] (10) 
where 
1 eee 
l+Vi-y lt+w 
Irom (8), we note that 
Ik ot 


aly) = WOOD. eae 


D9 


y manipulating (10) we arrive at the formula (valid 
fc 2) 


2y(1 — 
n+ 1 


y) d sae 


AnviY) = + ya,(y). (11) 
j. (11) lends itself readily to computation of p?(r). 

lor k = 3, the solution to the differential equation (9) 
volves Bessel functions (of 4 integral orders). For 
= 4, Weber functions are obtained; for k > 5, the 
netions are unnamed (and untabulated). In none of 
ese cases could we derive an expression analogous to (11). 


AsyMPToTic Property or p<(r) 


If all x, in X are independent, then in Z, z, is independent 
Zi+m, provided |m| > 2k — 1. The Morse distribution 
(r) then tends to the Gaussian’ for large n, and we 
ould expect that this happens sooner (with respect to 
e size of n) for smaller k. 

It remains, therefore, to find the first two moments of 
(r). With these, and (5) and (6), we will be in a position 
obtain a fairly accurate approximation to pi(r). 


Moments oF pi,(r) 


We shall prove that the mean and dispersion of r are 
ven by 


an Sie 1 DT 
(r) h (12) 


WK (b) 


_ __ 2k(5k + 3) 
(2k + 1h + 1) 


| N 2 2k (13) 


= 0,n + 1) (13’) 


To prove (12), we note first that E,(r) = >°2., rp,(r). 
rom 2(b) we can, with some manipulation, show that 


2¥ be) 


1=0 


i, = DE. 


E(r) = fae Ie: (14) 

We use (3) to establish 12(a), and prove 12(b) in- 

ictively, using (14). 

To prove (13), we note that.c. = Hr) = H,G))’, 

here E,(r°) = =, 1'p,(r) is the expected value of 7°. 

vain, from 2(b) we derive 
D) iat i 1 n—-1 

eh es Rise E. 
S do ari S| i(7) 


Joh. Olle Usa ne 


ho Ie (15) 


5 §. Bernstein, ‘Sur l’extension du théoreme limité de calcul des 
obabilités aux sommes des quantités dépendentes,” Ann. Math., 
1. 97, pp. 1-59; 1927. 
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If n > 2k, we can evaluate the second sum on tue right- 
hand side of (15). It equals 


Qk(k — 1) 


3(k 4 1) n({E,(r))’, 


and (15) becomes 
kk — 1) 
SB ey 


If we multiply (16) by n, add 20% and divide by n + 1, 
it becomes clear that 


n 2 2k. (16) 


=(0;, n > 2k CZ) 
where C;, depends only on k. We can now, for convenience, 
evaluate C, by evaluating o3,. We do this by going back 
to (15) and, after some lengthy manipulation, obtain 


o>, hence (13’). We have computed 


ee OC, — 2 mo _ 1097 
iar ty aati? a RIS TON)” 
and 
15611 
Co = 315315, 


By using the well-known function® 


I’) 
T(z) 


we can perform the sum appearing in (13), 2.e., 


¥@) = 


= ¥(2k + 2) — ¥(k + 2). 


It is known that 


og (2) + (4) 


for large z, and we thus obtain 


¥@) = 


(18) 


Tur Env Potnt PROBLEM 


The Z sequence is stationary between the limits 
ki Sy <n, —. bk, buisiors <9 ore) CVenntne 
individual probabilities that z; = 0 vary with 7. A com- 
pletely stationary sequence can be derived by omitting 
the end points. First, define a sequence U = (u;),7 = 2 — 
k n + k — 1, and a corresponding sequence V = 


eee as 


6 See ‘Higher Transcendental Functions,’’ vol. 1, sec. 1.7, 
formulas 1 and 10, and sec. 1.18, formula 7, Bateman Manuscript 
Project, McGraw-Hill Book Co., Ine., New York, ING Woes HOES, 
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(v;),4 = 2—k,---,n +k — 1, derived from U in the 
same way that Z was derived from X. The subsequence 
V? = (v),4 = 1p: , 1 will then be stationary. Let 
q.(r) be the probability that V’ has r zeros. 

We have succeeded in showing that 


Piaf + 1) = Gr) aw 2.0: (19) 


For k > 3, we have found no comparable equivalence. 
lor fixed k, it is clear that in the limit of large n, the end 
points assume less significance and p:,(r) and q,(r) approach 
each other. 


MEAN VALUE OF r FoR V’ 
Defining the mean of r in V’, E’(r) = 3524 rq,(r), we 
shall show that 


ee 


IO i= Helen pat 


(20) 

We first derive a formula for p(Z'),’ the probability 
of the event v; = 1. We shall discard the two restrictions 
that a) all terms of U are independent, and b) that they 
are statistically identical. They need merely have con- 
tinuous density functions. The formula is 


i 


Dy 


7=t-—k+1 


DE) = Soh aes 


jg=i-k+1 


p(B") = (21) 
where #&; is the event that w; is the largest variable in the 
Sequence U;, Ua; °° Ue, J KH Pe Le Oe 
2, --- ,7. By the fundamental theorem on the probability 
of the union of events’ we can write 


ph") = S; = So == S3 00 Se Sk 
where 


See pine So 


Si") phe): 


d<l<m 


el. 


the sums running from z — k + 1 to z. Now in our problem 
all probabilities are determined exclusively by the first 
and last indexes, plat (= phy lealy. a 


€.9., 
p(R RRR). Since (" i ') distinct product sets can be 


formed when there are 7 indexes between the first index 
and the last index 7 + n, and since 


Ss Gate 7 ') aif i 


r=0 


IV 
tb 


we alrive at (21). 


7 This derivation is due to L. Maximon “Some Properties of 
Infinite Sequences of Independent Random Variables,’ Group Rep. 
Nos. 34-53, Lincoln Lab., Lexington, Mass.; July 11, 1956. 

8 QO. O. Feller, “Probability Theory and Its Applications,” John 
Wiley and Sons, New York, N. Y., p. 61; 1950. 
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In the special case that all u; are independent an 
identically distributed, we have 


Les 


and pik; +3) a ae 1 


p(R;) = 


ale 


for all j. Hence by (21), the probability that 0; = 1 is 


1 are 
be A) re ee 


and so the probability that v; = 0 is 


i 2 aah 
EA Shea 


Eq. (20) then follows. 


1 


2 9 
ALTERNATE REcuRSION FORMULA FOR 7,(7) 


We have derived another recursion formula for pr) 


2 t= 1 2 n > il 
: 7 ) DaG = 1) Pe io 
Pp 2 ll 


2 DR os (Mae 
pilr) = = pia) +> (29) 


The proof is quite simple. Consider a sequence X, of 
integers from 2 to nm. Given that all 2; are statistically 
identical, if the integer | is inserted into the sequence to 
form a new sequence X,, this integer can appear with 
equal probability at any place 7 in X,. If Z, has r — I 
zeros, Z, must have either r — 1 or r zeros, or, alternately, 
if Z, has r zeros, Z, must have had r — 1 or r zeros. If the 


integer 1 is added adjacent to (on either side of) a mini- 
mum point of X,, no change occurs; otherwise one zero is 
added. If there were r zeros in Z,, which occurs with prob- 
ability p2_,(r), there is probability 2r/n of adding no 
additional zero, since, for k = 2, two successive zeros 
cannot occur. Similarly, for r — 1 zeros in Z, [probability 
p,:(r — 1)] there is probability [2 — 2(r — 1)]/n of 
adding a zero. Thus we get (22). 

This recursion makes it easy to compute successive 
values of p;(r). It is also possible to derive (11) from (22). 

For k = 3 it is possibie to write an analogous recursion 
formula for the joint density function p,(7,, 72), where 7; 
is the number of single zeros (those bounded by 1’s, 7.e., 
101) and r, is the number of pairs of zeros (1001). Since 
three successive zeros are impossible for k = 3,r = r; + 
2r., and p;(r) can be found by summing the p,(r:, 72) over 
r, and rz. We shall not reproduce this untidy formula here. 
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We list a,(y), with k indicated at the top of each list. 
The values of p’(r) can be obtained by comparing directly 
with (4). For example, p5(3) is the coefficient of y* in 
aey) under k = 2. Vhus. p,3)s—=—1/ 45: 


® This result was suggested by A. Kohlenberg. 
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k=2 
ay) = 1 
aly) = y 
ay) = y 
2 2 
a(y) = VEY 
aly) = ey 
3 
. 2y + lly? + 2y’ 
Qu + 26y? + 17y" 
as(y) = AR 
_. _ 4y + 114y’ + 180y* + 17y" 
aly) = 315 
one y + 60y° + 192y? + 62y* 
315 
Oye 2y + 247y + 1452y*? + 1072* + 62y’ 
ae 2835 
(y) 2y + 502y" + 5097y*? + 7192y* + 1382y” 
oe 14175 
an dy + 2026y? + 34096y’? + 83021y' + 35396y’ + 1382y/” 
po 155925 
ue Qy + 2036y> + 55196y’? + 217186y* + 171511ly® + 21844y" 
EN 467775 
4y + 8166y? + 349500y* + 2123860y* + 2801040y" + 776661y° + 21844y’ 
a3(y) = 6081075 
k = 3 
Aly) = 1 
a,(y) ac 
aly) = qe 
a3(y) = y 
2 ah ( 3 
as(y) =F - ae 
alt. 3y° Siig y 
a;(y) ae Z 5 
_ Qy? + ly’ + 17y* 
Aly) —— 30 
& Qy? + 17y*? + 65y* + 2ly’ 
OMe 105 


dy? + 24y> + 177y* + 196y’ + 21y’° 
as(y) = 420 
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dy’ + 64y*? + 819y* + 1934y? + 959y" 


oy) = 


3780 
7 dy? + 82y* + 1737y* + 7037y° + 8717y* + 1323y' 
aioly) = 18900 
dy? + 102y* + 8515y* + 21635y° + 51030y° + 26341y" + 1323y° 
aly) = 103950 
) 8y° + 248y° + 13894y* + 120149y° + 465379y° + 523031y" + 124691y° 
COM 1247400 
16y? + 592y* + 54392y* + 625222y° + 3639482y° + 7209496y’ + 4318450y* + 368550y" 
thaly) = 16216200 
k= 4 
ay(Y) = 1 
a,(y) ae Ui 
a(y) = y 
as(y) = y 
aly) = y? 
3 4 
ay) = ay oy 
9) 3 8 4 5 5 
ag(y) = “EAE OY 
4y*> + 28y' + 58y? + 15y 
a,(y) en 105 
Qy? + 2ly* + 78y’ + 109y° 
as(y) 4 210 
ope 2y*? + 29y* + 162y° + 526y° + 226y’ 
Ree 945 
One 4y’? + 76y' + 585y° + 3266y" + 4619y’ + 900y* 
Ee 9450 
(y) 8y* + 192y* + 1930y° + 16414y° + 46434y’ + 36272y° + 2700y’° 
ee 103950 
Ade) 8y* + 236y* + 2986y’ + 36533y° + 168787y’ + 288095y° + 127055y° 
ENS 623700 
(y) 16y’ + 568y* + 8804y*? + 150874y° + 1025054y’ + 2949718y*° + 3347326y" + 625740y"° 
Maes 8108100 . 
k=5 
aly) = 1 
a,(y) Sar 
a[y) = y 
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aly) = y" 
as(y) = y 
4 . D) Is 
a,(y) = boat 
4 @) 6 
a;(y) = 2y se oe => Oy 
Dijin 44a hae 
ae Yoo De Vie CA5) 
see 2y + 25y" + 112y" + 197y" + 42y° 
XY. 378 
ae 4y* + 68y° + 449y° + 1402y” + 1857y* 
Sa 3780 
ae 4y* + 88y° + 789y° + 3647y’ + 10757y* + 5505y° 
tA 20790 
aie Sy* + 220y° + 2546y° + 15973y’ + 75491y* + 122407y° + 32835y"" 
Ne Sag 249480 
Oe 16y* + 536y° + 7732y°® + 62498y’ + 417898y* + 1285346y° + 1289034y'° + 180180y"" 
Ys 3243240 
ke=30 
aly) = 1 
a,(y) == | 
a,(y) = y 
az(y) = y 
a,(y) = y 
Ey) = 
a6(Y) =a y 
5 6 
HOME 2y + Sy 
7 
y =e, 6y° ae Ty 
Aine y> + 10y° + 3ly’ + 2ly* 
Wf 63 
Qy® + 29y° + 152y’ + 321y* + 126y’” 
@yo(Y) = 
630 
+s ay? + 39y° + 297y’ + 108ly* + 1731y° + 315y"° 
aly) = 3465 
_ 4y’ + 100y° + 1023y’ + 5429y* + 15353y° + 19671y"° 
analy) = 41580 


4y? + 124y® + 1623y" + 11567y* + 47927y°? + 132579y"" + 76446y" 
a3(y) = 270270 


bo 
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Two Properties of Pseudo- 
Random Sequences* 


The so-called ‘‘pseudo-random’’ (p-r) 
sequences defined below have two properties 
which may make them useful in certain 
communication systems. Stated roughly, 
the first property is that the minimum 
distance between different signals of a 
special class is maximized and the second 
is that the probability that one member of 
this class be mistaken for some other 
member is minimized. 

Let Ao = (ai, G2, +++, Gn) be a sequence 
of length n in which a; = +1 ora; = —1 
and let A; be its k-th cyclic permutation. 
Thatis, Al, =" (Gpas, Gas, > nai, ae) 
for k = 0, 1, ---, m — 1. In analogy with 
Slepian’s! terminology, the set Ao, A1, °*-, 
A, shall be called an n-digit permutation 
alphabet if no two of the sequences A; are 
identical. The transmission of these se- 
quences over a symmetric binary channel 
in which the probability of erroneous 
reception of any digit a; is p, independent 
of errors in the other digits, will be the 
concern here. The detection of such se- 
quences may be considered to be a form 
of digital phase detection. Barker? and 
Sherman? have considered similar problems. 

Let r; be the covariance coefficient, 
defined by 


n 
yy Dn+ideritiy (1) 


t=1 


r; = (A, Agsi) = 


where Giin = @; and A;. Clearly, 


Aisn = 


» AiAi+;, (2) 


i=1 


for any value of k in (1). The sequences 
A; will be called ‘‘pseudo-random”’ if the 
covariance ee satisfy the relations 


ry = 7 
Po GG SD ey or ) 


The alphabet Ao, ---, An—1 will be called 
an n-digit p-r alphabet if (3) holds. The 
name “‘pseudo-random’”’ has been used by 
some authors‘ because of the resemblance 
of these sequences to white noise, as shown 
by (3). It should be noted that p-r sequences 


* Received by the PGIT, June 5, 
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do not exist for all values of n. In fact, 
it can easily be shown‘ that a necessary 
(but not sufficient) condition for the exist- 
ence of a p-r sequence is that n have the 
form 4m — 1, where m is an integer. Note 
that (2) and (3) do not specify the n 
quantities a; uniquely since the covariance 
coefficients are not independent. In fact, it 
is easily shown from (2) that 7; = 7r,_; for 
all j. 

Huffman’s® “maximum length null se- 
quences,”’ all of which are of length 2” —1 
(m an integer), have the p-r property 
[see (3)]. Barker® has given sequences of 
length 3, 7 and 11 which satisfy (8). Hence- 
forth, it will be assumed that there exists 
at least one p-r alphabet of the given 
length n. We will show that this alphabet 
is the best by two different criteria of all 
n-digit permutation alphabets. 

First, it will be shown that the mini- 
mum distance between different sequences 
of a permutation alphabet is maximized by 
choosing a p-r alphabet. As usual, the 
distance between two sequences is defined 
as the number of digits which must be 
altered to change the first sequence into 
the second. Since the product (a%4;)(@%4i+7) 
has the value +1-if diy; = Qnsiy; and 
otherwise has the value —1, it is easily 
seen that the distance between A, and 
Ax; is  (n— 7;). Let D be the minimum 
distance between any two sequences of an 
n-digit permutation alphabet. That is, 
D= min 4#(n—7,). (4) 

l<i<n-1 
For a p-r alphabet it follows from (3) 
that the minimum distance, D,, is given by 


Disa tak): (5) 
The following property will be proven: 


Theorem 1) The minimum distance for 
an n-digit pseudo-random alphabet is 
greater than or equal to the minimum 
distance for any other n-digit permutation 
alphabet. 

Proof: It is easily shown from (2) that 


n—1 


DB PEM Reh (6) 


j=l 
and hence that 


ww 


De Cas SAG sl) 


a 


ND 


where 
C= SY a;. (8) 


5 D. A. Huffman, ‘‘The synthesis of linear sequen- 
tial coding networks,”’ in ‘Information Theory, Third 
London Symposium,’’ C. Cherry, (ed.) Butterworths 
Scientific Publications, London, Eng., pp. 71-95; 
1956. See particularly p. 91. 

8 Barker, op. cit., p. 282. 
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Therefore, since 2D <n — rj, 
—%n—-)D<n—c¢ <n —1. OF 


The last step follows from the observation 
that c > 1 for odd n. Since a p-r sequence 
of length n exists, n is necessarily odd. 
Therefore, from (9) and (5), 


< 3(n + 1) = D,. 


This completes the proof. 

The minimum distance is one criterion” 
which might be used in evaluating error- 
correcting or other codes. Huffman’ has 
also discussed the error-correcting properties 
of maximum length null sequences irom 
another point of view. 

The second property of p-r alphabets: 
to be proven may be applied to systems: 
in which there is error detection but no. 
error correction. In such systems the 
probability of an incorrect decision is” 
minimized by the use of p-r alphabets. 

Theorem 2) The probability that a 
transmitted sequence of an n-digit perg 
mutation alphabet will be incorrectly 
identified as some other sequence of the 
alphabet is greater than or equal to the 
corresponding probability for an n-digtte 
pseudo-random alphabet. 

Proof: Since the number of digits which | 
must be altered to change A; to Ax,; is 
ae r;), the probability that A, will 
be identified by the receiver as some other 
sequence is | 


(10) 


: 
' 
| 
| 


a 


bs eel = Dee 


n=1 


P= (11) 


The values 7; which minimize P when these 
values are subjected to the constraint 
(6) are now determined. This problem 
is solved easily by the method of Lagrange 
multipliers. The result is that one must 
have 


r, = —(n—c)/(n — 1). 


When all the coefficients 7; are equal, it 
is clear from (11) that when p < # (which 
is assumed), P is minimized by choosing 
these coefficients to be as small as possible. 


Since @ > 1, the minimum is attained 
auidaveray Gay a il (ip Ih, +n = e 
Thus, P is least when 7, = —1G =a 
Ye OTS ip} 1), z.e., waen Ao, Ags 


is a p-r alphabet. 


L. Lornr CaMPBELL 
Defence Res. Telecommun. Establ. 
Ottawa, Ont. Can. 
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n Inequality Concerning the 
nvelope of a Correlation 
unction* 


_ The following inequality for a correlation 
nection (7) is proved: 


RO > (RO) + POI?) 


here, in terms of a “power spectrum’’ or 
vectral density w(f), 


| Rr) fs w(f) cos 2rfr df (2) 


rd 


io ened, a) 


This inequality, which is a stronger 
atement than the well-known fact that 
(0) > |R(r)|, is equivalent to the state- 
ent (hereby proven) that the envelope 
yroperly defined) of a correlation function 
a maximum at the origin. 

To prove (1), consider the following 
lantity, which may be thought of as a 
ymplex version of R(r): 


foo} 


[ w(fe?"*” df = Ones (4) 


Considering the integral in (4) and 
membering that w(f) is non-negative, we 
ay write, 


[ “wher” df | =}, “ wl) af (6) 


nee the absolute value of the integral 
ust be no greater than the integral of 
ie absolute value of the integrand. The 
ft-hand side of (5) is, however, 


Wien) teed (a) 


hereas the right-hand side is R(0). Thus 
e inequality (1) is proved. 

To demonstrate the connection of the 
equality with the envelope of R(7), the 
ivelope we define in the following way:! 
onsider the definition (4), in which Q(7) 
id 6&7) are taken to be real. Equating 
al and imaginary parts of (4) we obtain 


Q(7) cos [(7)] = R(7r) (6) 
id 


Q(z) sin [6(7)] = (7). 


'e obtain, further, 
OG) = {Rh (7) Tal (8) 


* Received by the PGIT, September 22, 1958. 
1 The envelope of R(r) is defined with the aid 
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dji, ‘‘Envelope and pre-envelopes of real wave- 
ms,” IRE Trans. ON INFORMATION THEORY, 
l. IT-9, pp. 58-57; March, 1958, where a number 
other references are cited. 
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where the positive square root is used. 
Then cos [@(7)] and sin [6(r)] are unambig- 
uously defined, from (6) and (7). It can 
now be seen by (6) that a correlation 
function R(r) can be written as the product 
of Q(7) and cos [6(r)], where Q(r) and 
cos [@(7)] are unambiguously defined and 
where |cos [6(r)]| < 1. Q(7) is now called 
the envelope of R(r). This cognomen is 
seen to be natural and convincing. One 
may verify by specific calculation that the 
envelope as defined by (8) coincides, at 
least In many cases, with what one would 
consider the ‘“coarse’’ structure of (7) 
As an example of the fundamental nature 
of this envelope Q(7), one may point to a 
recently published paper,? in which it is 
shown that the output of a “band-pass 
correlation detector’ is what we have 
called Q(r). 

Since Q(0) = R(0), the inequality (1) 
is now seen to state that O(0) > Q(7), 
7.e., the envelope of a correlation function 
has a maximum at the origin of 7. This 
is stronger than the statement that R(O) > 
|R(r)| since J? (7) is non-negative. 


Puiure R. KARR 

Ramo Wooldridge 

Div. of Thompson Ramo-Wooldridge Inc. 
Los Angeles, Calif. 


2P. E. Green Jr. “The output signal-to-noise 
ratio of correlation detectors,’ IRE Trans. on 
INFORMATION THEORY, vol. IT-3, pp. 10-18; March, 
1957. 


Lossless Symbol Coding with 
Nonprimes* 


Golay! has asked the question as to 
‘““whether the search for master iteration 
matrices, of the form p” for values of n 
higher than 2 can be systematized.”’ There 
exists a straight-forward method for con- 
structing the coding matrix for realizing all 
the one-error correction codes whose exist- 
ence was shown by Zaremba.? The ability to 
do this depends upon the fact that there 
exist finite fields of order p”, where p is a 
prime, namely the Galois fields. The method 
is completely constructive since in order to 
construct the Galois field GF(p”) it is only 
necessary to find a polynomial of degree n 
which is irreducible over the field of integers 
mod p. This is constructive since there is 
only a finite number of these polynomials; if 
one can’t find a better way to obtain an irre- 
ducible polynomial, one can test each 
of these polynomials for irreducibility. 
Assuming that a Galois field of order p” 
has been constructed, the following method 


* Received by the PGIT', January 26, 1959. 
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is one way of obtaining the coding matrix. 
Incidently, the method proves construc- 
tively the theorem of Zaremba as to the 
existence of such codes. 

Let 0, 1, a3,°°*, Gpn be the elements of a 
Galois field Gi'(p") and H be the prime 
ideal defining said field. Suppose one wishes 
to construct a single-symbol correcting 
code with k check symbols. First write 
down all possible distinct column vectors 
v; of length & deleting the vector of the 
form 0, and, although this is not necessary, 
make the first k vectors of the list of the 
form v; = 6;; (kronecker delta); this sim- 
plifies encoding and decoding. One has 
p"* — | distinct vectors. Now systematically 
delete from the list all vectors which are of 
the form v; = av; fori < j and ae GF(p”). 
Since a field has no zero divisors, one 
clearly has left p™* — 1/p” — 1 vectors. Now 
construct a code with s = p** — 1/p” — 1 
total symbols, & check symbols, and s — k 
information symbols. Let A be the matrix 
composed of the vectors left after the 
deletion process. The column order is the 
same as left by the processes. 

Given s — k information symbols, form 
a column vector w of length s, the first k 
elements from the top being 0 and the 


remaining s — k elements composing the 
given information symbols. Forming 
Aw = z mod (H), one obtains a column 


vector of length k. Since the elements of the 
vector are in a field, the additive inverse —z 
of z exists. Form a vector x which has —z 
as its first k elements and the information 
symbols as the remaining s — k elements. 
Since the first & columns of the matrix are 
of the form v; = 6;;, we see that Ax = 0 
mod H. This is now the encoded message. 
Suppose there is an error of amount b in 
the eth character. Then the received message 
isz + 6.;b. Form A(x + 6.;b) = Aé.;b =v.b 
mod (ff). 

Looking at v-b we can determine that 
the eth character must be in error, for by 
construction of A, v.b is not of the form 
vja for any 7 # e and a e GF(p”). If this 
were so we would have v.b = vj;a or vp. = 
vjab, and v. would be the multiple of a 
vector v;; and this is not the case by con- 
struction. Thus we know that the eth 
character is in error and, further solving 
vy = vb, we can obtain y = b and thus 
correct any single symbol error. It might 
be noted that this is as far as this particular 
method can be extended since we are 
relying on the property that the elements 
form a field and every finite field is iso- 
morphic to some Galois field. 

In order to determine the Master 
Iteration Matrices (MIM) so that one 
can check the codes using the multiplication 
table of the integers mod p rather than the 
multiplication table of the Galois field, the 
following procedure is easily seen to work. 
Each element of a field GF(p") can be 
represented as a polynomial of the form 

29 ciz* where c; are the integers mod p. 
Make the correspondence ¢ = 072) cv*< 
(Co, C1 *** Crna) = c’ making this a column 
vector. If a « GF(p") is a typical element of 
the coding matrix A, form the product a-c 
where a = 72) a;z? and the a; are the 
particular coefficients in a, and ¢ = 


) 
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So"=) cw’ where the c; are indeterminates. 
Then using the polynomial H to reduce all 
the powers we determine a matrix a* 
which if 


a 


v 
os. Dish SRO BO 


i=0 


rs) Se 


Sa Doe 
iy 08 Cha) = Ol 


then a*c’ = d’. Thus matrix a* is the MIM 
searched for. Replacing the elements of A 
by the corresponding MIM we obtain a 
new matrix A’ which operates on vectors 
where each element of the message is repre- 
sented by a column vector of the form 
(co *** Cr+) where the c¢,’s are integers 
mod p. This matrix is used to encode the 
message as before but the congruence is 
mod p instead of mod H. 

Example: Let 0, 1, x, « + 1 be the elements 
of GF(2?) where H = x? + x + 1. Then 
let k = 2; we calculate s = 5; a possible 
matrix turns out to be: 


Wi OF lle 
OAPI al 


a+1 
ae al 


To form the MIM we see that 1-(c% + co) = 
cit + co, so 1 gives a and (a + 1) 
(ce + co) = c1(a? + &) + cz + Co = (40) + 
Co + a, so (x + 1) gives ‘Gay Thus the 


matrix becomes 


E020 Oe is 0.91 1 
OP Oe Oe 0em ea O RL 0 
OR ORK i's als Os sett i salamat) 
Ose Orr Oates Leet 1 


One thing that is interesting is that the 
Hamming Code has lossless points at 
S74 = Mh FB fy Sy OR 4 9 co JG 
the lossless codes of radix 2? we obtain 
s = 4*1/3 = 1, 5, 18 ... or, since each 
character is 2 bits, it is bitwise 1, 26,.... 
and the number of check bits would be 
2, 4, 6. Comparing this with the Hamming 
code, notice for instance that for s = 10 one 
must use 4 bits so that what this code does 
to becone lossless is to correct 5 double 
errors. Thus for radix 2? and assuming that 
the errors are not correlated and that all 
have equal probability we are able to con- 
struct some best possible codes. This, of 
course, can be done easily by the proper 
interpretation of the Hamming check bits. 
However, it interested me that this code 
did it automatically. 
JOHN CocKE 
IBM 
Poughkeepsie, N. Y. 


A Comment on a Comment on 
Pattern Redundancy* 


In a recent letter! concerning an article 
by Glovazky,2 Lowenschuss gives a set of 
necessary constraints on the choice of a 
code schedule for distinguishing  over- 
specified black and white patterns. These 
constraints are in fact not necessary except 
in the case that P = 2”. This can be seen 
from the following simpie counter example. 
We have nine patterns specified by the 
four bit codes given in Fig. 1. In the ter- 
minology of Lowenschuss this would make 
C = 4, P = 9. The constraints derived 


using Lowenschuss’ necessary conditions 
are 

AR SCy OD) 

Wee SE 

4 <6, < & 

I< <8 


Or, in other words, three of the columns 
chosen should have either four or five “‘ones.”’ 
Examination of the number of ones in 
the columns of Fig. 1 reveals that there is 
only one column which meets this condition. 
According to Lowenschuss, separation with 
four cells would then be impossible. How- 
ever, the four cells given do result in separa- 
tion (no two of the rows of Fig. 1 are the 
same). 


KS 
= 
‘\. Columns 
~ 


\ 
IN 
Patterns \. 


1 1 1 0) 
2 1 1 1 
3 0] 0 1 
4 0 1 0 
5 0) 1 1 
6 1 0 0 
fd 1 0 1 
8 il 1 0 
9 1 1 1 
Column Sums— 6 6 5 
Fig.—1 
Actually Lowenschuss’ necessary con- 


dition is applicable if separation must be 
obtained into consecutive codes. That is, if 
the only possible correct choices of cells 
are those which yield a set of binary codes 
ordered according to their numerical value. 
This condition, however, is not necessary 
for separation. 
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vol. IT—4, p. 127; September, 1958. 
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One can obtain a generalization of the 
constraint given by Lowenschuss for P = 2nd 
The procedure is as follows: Suppose we 
are given P codes and are attempting to 
affect. separation with C cells. Then we can 
derive a constraint on the total number of 
ones in all C cell columns. All 2° C-bit 
binary numbers are listed; first the binary 
number having C ones, followed by all 
those binary numbers having C — 1 ones, 
then all those having C — 2 ones and so 
forth. It follows that if we count the number 
of ones in the first P binary numbers of 
this listing, the result will be an upper 
bound on the total number of ones allowable) 
in all C cell columns. Similarly we can 
derive a constraint on the total number of 
ones in any C — 1 of the m columns. All 
2¢ C-bit binary numbers are listed; first 
all the binary numbers with C — 1 ones in 
the first C — 1 columns, then all those. 
having C — 2 ones in the first C — 1 
columns and so forth. It follows that if we 
count the total number of ones in the first 
C — 1 columns of the first P binary numbers 
of this listing, we have an upper bound on 
the total number of ones in any C — 1 of 
the C cell columns chosen for separation. 

Similarly upper bounds can be derived 
on is Sue of ones in any k columns | 
(a - C). The calculation of these 
upper onan is summarized symbolically 
below. 


(*) = Number of combinations of “kia 


ar) 29) 


things taken at a time. 


P = Number of ee 
C = Number of separation cells. 7 
1 = Upper bound on the number of ones” 
in any k of the C separation cells. | 
i=k k | 
Th eats rool ) : 
i=0 v | 
where 
fc® = Minimum of . 
iP 
1, and i 
( Voge 
k 
and 
f:* = Minimum of 
Pia y iL tb ; 
1, and jaitt 
a) 
a 
tore iy 22%. 


As an example let P = 11, and C = 
Then we find that | 

T! = 8 > number of ones in any single 
column 

T? = 15 > number of ones in any two 
columns 

7% = 21 > number of ones in any three 
columns, and 

T* = 28 > number of ones in all four 
columns. 


ow we can get constraints on individual 
umns. If ¢; is the number of ones in the 
cell column, and we agree to order the 
o that if ¢ < j then ec; > c; we can 
vrite the above constraints as follows: 


Cer 
Om Coe = Lp 
Cie Osi 03 21 


Creer iC, = Ca 8. 


ving these for the individual c; we obtain 
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In general we can express these individual 
column constraints as 
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where [ ] means the next lowest integer. 
Furthermore it can be shown that 77/7 < 
T’/j — 1, so we have finally 
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The upper bounds on the number of 
zeros in the C cell columns are the same as 
those for the number of ones, and can be 
used to get lower bounds on the number of 
ones in the cell columns. 

The calculation of the constraints 
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calculation would seem worthwhile if 
efficiency was of concern in determining a 
code schedule. 
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